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ABOUT THIS PROJECT 



A Case Study of US Education 



It is widely acknowledged that innovation will be necessary to 
dramatically improve public services in America. But innovation 
in the public sector doesn’t happen in a vacuum; innovation 
happens at the nexus of policy, research, capital, and practice. This project looks at one 
case study - education - by analyzing some of the key aspects of an emerging ecosystem for 
innovation in public education in the US, including the flow of investment capital for such 
efforts, the uptake of innovations by buyers and users, federal efforts to stimulate and scale 
innovation, and ways that technology could facilitate innovation investment and practice. 
Drawing on surveys, interviews, and working groups, the project highlights recent efforts to 
fuel and steer more innovation, and frames the remaining challenges that lie ahead for the 
public, private, and philanthropic sectors. This project culminates in an analysis of the lessons 
and insights drawn from the recent experience of US public education in comparison to the 
way leaders are using innovation to address similar intractable social problems in other fields 
and in other countries. 



For more on this project and its publications, visit http://www.bellwethereducation.org/ 
innovation-for-the-public-good/. 
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INTRODUCTION 



We are at a critical inflection point in America. The global knowledge age asks more of our 
schools than ever before — that they educate all students and do so to a much higher level. But 
student achievement is mediocre when compared to international benchmarks, and downright 
appalling in the urban and rural areas that serve most low-income and minority students. We 
are far from providing our children with universally high-quality public education — the kind 
that equips all students with the knowledge and skills they need to secure a college degree, 
earn a good living, support a family and engage productively in society. 

To change this equation, and to do so without a massive injection of new resources, we 
clearly need to do things differently — and fast. Most acknowledge that the solution to 
this productivity challenge is innovation. But innovation doesn’t emerge out of thin air, 
especially in a public field like education. It emerges from the efforts of visionary people 
and organizations that try to do things differently, despite being surrounded by people and 
organizations doing things the way they always have. Accordingly, an innovation ecosystem 
includes not just the innovators themselves, but also users, buyers, investors and researchers — 
and in the case of a public good like education, policymakers and the rules and regulations 
they establish, which can either encourage or hinder these new ways of operating. 

This nexus — where a large-scale social problem meets innovation, capital and public policy — 
is the subject of the “Innovation for the Public Good” series, led by Bellwether Education 
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Partners and supported by the Rockefeller Foundation. This project considers a variety of 
efforts to improve the public education innovation ecosystem in the United States, as a case 
study that may have implications for improvement in other public goods. This particular 
paper seeks to assess the initial effect of the first round of the U.S. Department of Education’s 
Investing in Innovation (i3) initiative on specific elements of the innovation ecosystem in 
education, including the private and philanthropic sectors and the work of innovators 
themselves. 

First established as part of the American Recovery and Reinvestment Act (otherwise known as 
the “stimulus” legislation), and now continued as a regular Department program in the fiscal 
2012 year, i3 was intended to do three very ambitious things at once. 

First and foremost, the Department seeks to shift the dynamics in the innovation ecosystem 
toward greater quantity, rigor and diffusion of innovation, with the belief that this could 
improve educational outcomes for underserved students in America. At its core, the program 
created a stream of federal funding directed at increasing and strengthening innovation in 
public education — a part of the public sector that has long resisted the risk-taking, agility, 
speed and continuous improvement that come with embracing innovation. Perhaps equally 
important, it also explicitly sought to avoid the “islands of excellence” phenomenon long a 

problem in education and other social sectors, where 
innovations rarely manage to reach meaningful “scale.” 

Second, the Department is attempting to thread innovation 
into the way its own processes worked. The i3 fund is a 
far more evidence-based and transparent way of allocating 
resources than is typical of federal education funding. It 
also introduced to education the “field scan” mechanism — a 
common tool in the sciences, where innovation thrives — 
which is designed to reveal existing innovations in the 
field rather than dictating strict top-down parameters for 
government funding. This “field scan” was part of a deliberate 
multi-pronged approach to promoting innovation that included improved research and more 
intentional development of research findings into tangible products and programs (see sidebar 
on page 4). “In a sector as large as education and learning, there are myriad different efforts 
and approaches to improve student learning being tried at any given time. Inevitably, some 
will demonstrate significant promise while others will fail to meet existing needs,” notes a 
concept paper on ARPA-ED, the piece of the Department’s innovation initiative designed to 



"Field scans of efforts by 
practitioners and others 
throughout education can 
help identify and support 
the most successful, ideally 
resulting in the spread of 
effective ideas." 

-U.S. Department of Education 
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intentionally develop research into usable tools, which was announced in early 2011. “Field 
scans of efforts by practitioners and others throughout education can help identify and 
support the most successful, ideally resulting in the spread of effective ideas.” 1 i3 also aligned 
funding for the “supply” of innovation with the “demand” the U.S. Department of Education 
had created through other federal programs. By articulating goals and priorities that were 
aligned with the problems of practice identified by state Race to the Top applicants, and then 
soliciting the field’s best ideas for how to provide more innovative solutions to those problems, 
i3 tightened the link between good ideas and the buyers and users that would put those ideas 
into practice. 

Finally, as a public-private partnership, the i3 program explicitly seeks to create greater 
alignment among innovation investments in the public, private and philanthropic sectors, and 

to improve the quality and sustainability of these efforts. The 
program asked the private sector (especially philanthropy) to 
rally behind a set of federally identified innovation priorities, 
to take scale and impact seriously, and to allocate more 
funding for more evidence of success. Though not explicitly 
built into the design of the i3 program’s first round, the pace 
required by stimulus funding pressed the federal government 
and philanthropic donors to act quickly, which forced them to 
shake up their usual idiosyncratic decision-making processes. 

Taken together, these efforts constituted a significant federal 
investment in shaping and strengthening the ecosystem for 
innovation in U.S. education — though not without some 
challenges along the way. Less than a year after the first-ever 
i3 grants were disbursed and in the midst of the program’s 
second round of applications, it’s obviously far too soon to 
tell whether i3 has led to a greater quantity or better quality 
of innovation, or even to significant improvements in the way 
philanthropy operates — let alone whether student achievement 
will rise as a result. But it’s never too soon to begin the learning cycle by getting some early 
perspective about the program’s initial effects on the ecosystem in the eyes of key stakeholders, 
including grantees, applicants not selected for funding, philanthropic donors and the federal 
agency itself. 

To do so, we began soon after the first round of i3 was complete with a survey of more than 
300 people, including i3 applicants and philanthropists, which had a 20 percent response 



It's obviously far too soon 
to tell whether i3 has led to 
a greater quantity or better 
quality of innovation, or even 
to significant improvements 
in the way philanthropy 
operates — let alone whether 
student achievement will rise 
as a result. But it's never too 
soon to begin the learning 
cycle by getting some early 
perspective about the 
program's initial effects on 
the ecosystem. 
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rate. We also reviewed the extensive set of documents made publicly available by the 
Department about the process, as well as the relatively limited amount of i3 analysis to date 
from the education reform community and in the media. Finally, we also conducted nearly 
50 interviews with diverse stakeholders, including a selection of applicants (grantees and 
those not selected), key players in government and philanthropy, and other observers in the 



The Federal Cycle of Innovation in Education: 
Research, Development and Dissemination 



Just as innovation spans well beyond an initial "spark" 
of ingenuity, the i3 effort stands as one piece in a 
series of interrelated efforts by the U.S. Department 
of Education to create a more robust innovation 
ecosystem for American public education. 

As we explained in a recent paper, the cycle of 
innovation has many stages, including what is often 
referred to as research and development (R&D). 2 
The research phase includes both basic and applied 
research designed to develop knowledge, and is 
followed by development, which could be intentional 
or "directed" development or ways to encourage 
field-based developments, as i3 sought to do. 

In an effective system, during the development 
phase, promising innovations are built, tested and 
refined quickly in order to identify those with the 
greatest potential for impact, eventually leading to 
efforts to take effective innovations "to scale" by 
distributing them more widely. While the process 
doesn't always follow these precise steps, the most 
effective approaches to innovation pay heed to 
research, development, iterative evaluation cycles 
and scale. And, as innovation scholar Amar Bhide has 
noted, in effective systems, users and buyers often 
"play a venturesome or 'entrepreneurial' role in the 
design of new products, bearing 'unmeasurable and 
unquantifiable' risks and developing ground-level 
knowledge." 3 But in education, this adoption dynamic 



has been notoriously weak, and is one of the factors 
that has inhibited the development and spread of 
innovation. 

While the U.S. Department of Education has 
historically underinvested in this ecosystem, or 
occasionally addressed some parts but in a piecemeal 
way, the current administration appears to be 
addressing multiple parts of the education innovation 
ecosystem with these efforts: 

Institute of Education Sciences. Most of the federal 
resources for educational research flow through the 
Institute of Education Sciences (IES), but a common 
criticism of the agency has been that its findings rarely 
translate into meaningful improvements in practice. 

In late 2010, IES approved a set of new priorities that 
includes ways of better aligning its resources and 
knowledge with the greatest needs in the field (by 
seeking "to identify education policies, programs, 
and practices that improve education outcomes, and 
to determine how, why, for whom, and under what 
conditions they are effective") 4 and building local 
capacity to gather and use evidence. IES is looking 
for ways both to encourage districts to conduct their 
own low-cost evaluations 5 and to design research 
that allows the context of the intervention to be 
examined 6 ; it also recently redesigned the "What 
Works Clearinghouse" Web site to make it easier for 
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ecosystem. Because all surveys, research and interviews were completed before the second 
round of i3 was announced in June 2011, our analysis focuses exclusively on the first round 
of applications and grantees. (See page 56 for a consideration of the changes made in the 
program’s second round.) 



visitors to sift through applicable research findings. 7 
In addition, the institute is pursuing innovative new 
approaches to field-based practical research, like 
the new 90-day research cycle being tested by IES 
in partnership with the Carnegie Foundation for the 
Advancement of Teaching. 8 

Race to the Top. Though not an R&D investment per 
se, the federal "Race to the Top" (RTT) competition 
sought to intentionally and substantially improve the 
"demand" dynamic for innovation in education. By 
providing incentives for states and districts to remove 
barriers to performance-driven practices, Race to 
the Top generated greater demand for innovative 
solutions such as better data management tools 
and new and better curriculum and assessments. In 
particular, the emphasis on creating common state 
standards — an effort already under way that was 
accelerated by RTT — has removed a significant and 
decades-old barrier to innovation in tools, assessments 
and content, a barrier caused by requiring new 
providers to tackle 50 different sets of state standards. 

ARPA-ED. In early 201 1 , the Department introduced 
its ARPA-ED initiative to foster the kind of intentional 
"directed development" that its namesake, the 
Defense Advanced Research Projects Agency 
(DARPA), has contributed to the Department of 
Defense. "Directed development provides the ability 



to pursue a small number of high-impact projects, 
from concept through demonstration or prototyping," 
notes an ARPA-ED concept paper published by 
the Department. "Directed development projects 
begin with a specific end goal, rather than the aim 
to increase broad areas of knowledge, and generally 
include a defined time period and path forward." 9 
At DARPA, early basic research is targeted at specific 
problems of practice in the defense community, and 
can be intentionally and rapidly tested and refined 
thanks to close partnerships with defense customers. 
Similarly, the president's FY 2012 budget request 
included $1 2 million for ARPA-ED to "fund projects 
performed by industry, universities, or other innovative 
organizations, selected based on their potential 
to create a dramatic breakthrough in learning and 
teaching." 

Digital Promise. In 2010, Congress finally authorized 
funds for the National Center for Research in 
Advanced Information and Digital Technologies, 
also known as the "Digital Promise," which had been 
established in 2008. As an independent nonprofit, 
the Center will draw funds and oversight from across 
the public, private and nonprofit sectors in order 
to "harness the increasing capability of advanced 
information and digital technologies to improve 
all levels of learning and education, formal and 
informal." 10 
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At minimum, it is clear that the first round i3 directed significant attention to the need for 
more productive innovation in education, contributed significant resources to the work of 49 
winning applicants, and mobilized almost $800 million toward a coordinated set of priorities 
designed to improve public education, including nearly $650 million in federal dollars and 
another $140 million in matching funds from the private sector. One straightforward piece of 
our analysis will be a simple breakdown of the i3 applicants and grantees in the first round 
of i3, as well as the private monies allocated to them, to shed light on where these funds are 
focused and the kinds of organizations and innovations involved in the process. 

In this paper, we will also examine the design and intent of the original i3 program, and the 
ways in which the Department carried it out — including the progress that was made, but 
also the opportunities missed. In addition, because i3 was meant to have an impact on the 
innovation ecosystem by influencing and steering the actions of others, the perception of the 
program’s impact is as important to consider as its direct impact. Therefore, we will convey 
the mixed opinions we heard among those that i3 sought to influence — including education 
organizations and capital providers — about whether the Department of Education supported 
innovation in a meaningful way and how the first round of i3 affected the wider landscape. 

We will share observations from the field on how i3 has pushed the Department and the 
innovation landscape forward — by prioritizing student outcomes as the goal for and the metric 
by which innovations should be judged, establishing a staged approach to evidence and scale, 
emphasizing the importance of scale as a goal for innovation, and encouraging connections 
across sectors and between districts and independent nonprofit organizations. We will also 
convey the concerns of skeptics, who raise questions about the design and implementation of 
the original i3 program, who wonder if groundbreaking innovation can really be found in the 
ranks of its grantees, and who question the degree to which it mobilized private capital toward 
more productive innovation. And we will share some information about the second round of 
i3, announced just prior to the release of this publication, which has been adjusted to address 
some — but not all — of these concerns. 

Synthesizing the reflections from fans and skeptics alike — along with publicly available 
information and our own experience in the field — we will assess implications from the initial 
round of i3 for how the government fuels and funds innovation in education, and how 
it structures public-private partnerships toward this end. We will conclude with our own 
recommendations for how the U.S. Department of Education and others could continue 
to improve i3 in future rounds and other ways to advance and strengthen the American 
innovation ecosystem in education. 
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ON INNOVATION IN PUBLIC EDUCATION 



In America’s public schools, student attainment and achievement are simply not at the level 
necessary for success in the knowledge age and global economy. Access to a quality public 
education is a deeply held American value, and taxpayers spend more than $500 billion per 
year to accomplish it. But few believe we have succeeded. Although we have racheted up our 
demands for public schools, asking them to prepare an increasing number and diversity of 
students for success in college, the public resists allocating even more money toward a system 
that doesn’t seem to be achieving enough with existing resources. A poll conducted in 2010 by 
Education Next and Harvard’s Program on Education Policy and Governance (PEPG) found 
widespread pessimism about school quality, with one-quarter grading America’s schools as a 
“D” or “F”; however, while 63 percent of respondents favored an increase in “government 
funding for public schools in your district,” only 29 percent would support a local tax 
increase. 11 

This is a classic productivity problem. We need to do more with less. No one believes this 
can be accomplished simply by asking educators to work harder. And few believe the current 
system is set up to achieve these goals. Most efforts to date to reform public schooling 
have tinkered around the edges rather than redesigning the system in accordance with 
these ambitious outcomes. But new demands call for entirely new ways of structuring and 
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overseeing education, coupled with dramatically different approaches to channeling funding 
to schools and learning, new mechanisms for preparing teachers and leaders, and radically 
different models for delivering instruction and measuring achievement. 

Enter innovation. As we noted in our recent paper “Steering Capital” (also part of the 
“Innovation for the Public Good” project) 12 , while innovation often connotes exciting, shiny, 
brand-new and wildly different, all innovation really means is new ways of doing things that 
bring about an improved result at scale. Sometimes those innovations look quite familiar 
and other times they feel entirely new and unique. As innovation writer and professor Clay 
Christensen describes, this is because some innovations are “disruptive,” breaking with 
current practice to serve a new customer base or to serve an existing population in radically 
different ways, while others are considered “sustaining,” making improvements within the 
existing architecture of the current system. Education needs more of both of these kinds of 
innovations. 

Innovations also take different forms, as we have identified in a prior paper on the subject . 13 
Most think naturally of product innovations like the personal computer or a new 

pharmaceutical, but some innovations come in the form of 
processes (such as the simple hand-washing protocol that has 

as the “app” platform that has enabled thousands of 
developers to create modular software products for iPhones 
and iPads). Some observers, like Judith Rodin of the 
Rockefeller Foundation (which supported this research 
project), identify a special kind of platform innovation as 
“organizational innovations,” but the concept is the same: a 
new way of organizing resources and structures, including a 
new standard that influences how related products and 
process innovations emerge. Rodin also adds to the list 
“market innovations,” explaining for example how 
Rockefeller has attempted to create better conditions for 
“impact investors” to obtain both financial and social returns by setting up a network for 
those investors to work together but also coordinating entrepreneurs who receive that 
funding . 14 By addressing the way different people and organizations interact to create value, 
market innovations push past the old ideologies about the role of governments and markets 
and can mobilize and organize resources, people, ideas and institutions. 



saved millions from hospital infections) and platforms (such 
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Based on our research and observations, we would posit that the Investing in Innovation (i3) 
fund represents an attempt to identify field-based product and process innovations — while 
itself demonstrating an early-stage organizational and market innovation within government. 
Through i3 and other related efforts (see sidebar on page 4), the U.S. Department of Education 
appears to be attempting to transform itself from a bureaucratic, compliance-based federal 
agency into one that uses its unique position in the ecosystem (size, cross-state role and ability 
to drive national influence) to foster a stronger ecosystem for innovation. 

This notion of an “ecosystem” is crucial, because in order for innovation to thrive and grow, 
it must take into account a wide range of stakeholders and the ways that they work together. 
This includes the typical players in schooling, including students, parents, teachers and 
administrators, but also those often considered only at the margin, like institutions of higher 
education that prepare teachers and leaders, nonprofit and for-profit suppliers of products 
and services for schools (including large education publishers and entrepreneurial start-ups), 
the private and philanthropic capital providers that provide funding to start and grow those 
organizations, and the policymakers. These policymakers define the goals and operating 
conditions of public education through laws and regulations (at the federal, state and local 
levels), provide operating capital to schools and related institutions (the vast majority from 
the state and local levels) and occasionally provide some investment capital for new initiatives 
(primarily at the federal level). 

Like any other ecosystem, the educational ecosystem must also change and adapt as the needs 
of its participants change — which creates great friction in education. While constant change is 
considered unavoidable in most other parts of our lives, within education many resist it, and 
often seem to believe that the full “supply” of tools and talent and ideas needed for public 
education to thrive in this new global knowledge age already exist within the system and its 
schools, within higher education programs, and in the methods and products already available 
to teach students and manage schools. Compared to other fields of similar scale and import 
(like medicine or energy), there has historically been real resistance in education to the idea 
of improving or increasing the diversity of supply of any of these resources, especially when 
doing so threatens the historical dynamics in the ecosystem or implies that useful knowledge, 
ideas, people and resources might lie outside of the current providers. This inward-looking 
dynamic oversimplifies the problem, and inappropriately narrows the solution set. 

Increasing innovation in education calls upon all these stakeholders to change their behavior 
in meaningful ways, and to do so over time in a cycle of continuous improvement. As 
countless innovation researchers have described, the crucial element of an effective innovation 
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is not just the spark of “new” or even the “better” of the initial results, but the way in which 
it leads to a continuous learning and improvement cycle within the ecosystem that allows ideas 
to feed off each other, improve one another and multiply. This is no less true when it comes to 
evaluating the initial round of i3, as former U.S. Department of Education official John Bailey 
pointed out when the first round of grantees was announced. He wrote: 

Innovation investments, while needed, are insufficient by themselves. More attention needs 
to be paid to creating the conditions in which innovation can occur - what some refer to 
as an innovation ecosystem that includes human capital, financial capital, and regulatory 
environment. It isn’t that there is a shortage of ideas or entrepreneurs in the education 
sector, it is that the tangled web of regulations, entrenched bureaucracies, and outdated 
policies [makes] it difficult for any of these innovations to be adopted by traditional 
schools stuck in their traditional system . 15 

The premise of this paper is that given the need to improve public schooling across the U.S., 
the federal Department of Education has a unique and important role to play in creating and 
sustaining a cycle of innovation and learning for public education. The core questions we shall 
consider are: to what degree did the initial i3 program make progress toward that goal, and 
what can we learn from that experience to continue advancing innovation in public education 
for the benefit of the millions of students who rely on the public school system every day? 
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THE INVESTING IN INNOVATION PROGRAM 



Laying the Foundation for i3 

The mission of the U.S. Department of Education, according to its Web site, is “to promote 
student achievement and preparation for global competitiveness by fostering educational 
excellence and ensuring equal access.” It does this primarily by distributing federal education 
funding and creating policies that dictate how it will be spent and monitored, although it 
also collects data, commissions and disseminates research, and makes sure nobody using 
federal funds is violating discrimination laws. 16 While the Investing in Innovation Fund comes 
primarily out of that first bucket of work — putting money to work, as the name implies — it is 
also an example of how the Department focuses national attention on key educational issues 
of national importance: the achievement gap and declining student achievement relative to 
other countries. 

Over the last decade, the law that has set the tone for the way the federal government has 
worked with states, districts and schools — particularly in low-income areas — is the No Child 
Left Behind (NCLB) Act, which was enacted in 2002 as the latest version of the Elementary 
and Secondary Education Act, the primary federal education funding law since the 1960s. 

No Child Left Behind mandated that schools receiving federal funding make annual progress 
toward moving all students to proficiency, as demonstrated by gains on standardized state 
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tests, and that schools “disaggregate” those scores by student characteristics like income and 
ethnicity. It also imposed escalating penalties for schools and districts that failed to make 
progress. Among other things, NCLB is widely credited with placing a continued emphasis on 
holding schools accountable for improving student achievement, and on shining a spotlight 
on the needs of groups of students that had been underserved — whose performance had been 
masked when only average school scores were reported. 

Still, NCLB is far from perfect, and many saw an opportunity to improve upon it when the 
act’s congressional reauthorization appeared on the horizon in 2007. One relevant piece of 
legislation introduced during that time — while George W. Bush was president — was the 

Bringing Success to Scale Act (HR 3611), whose echoes would 
be felt several years later in the language surrounding the 

to establish awards for local education agencies (LEAs) and 
schools that demonstrated academic achievement and formed 
partnerships with the private sector that would provide 
matching funds to bring results to scale. Notably, the bill 
would also have awarded grants “to school management or 
support organizations, nonprofit organizations, and human 
capital providers so that they can work in partnership with 
the private sector and philanthropic community to expand 
innovative programs that produce results and share best 
practices with schools and LEAs.” 17 The 2007 legislation specified that these organizations 
could include school management organizations, human capital providers, educational tools, 
but also those supporting parental and community involvement and those providing before- 
school, after-school, or in-school academic, emotional and behavioral supports. “The 
mechanisms to evaluate impact on academic achievement already exist,” noted Charles Harris, 
a longtime investment banker who founded SeaChange Capital Partners to provide growth 
funding to education organizations, in his testimony at the time. “I believe it is critical for the 
federal government and the private sector to collaborate to identify what works and to bring 
the substantial financial resources required to the table.” 18 

Such a competitive program would not have been entirely new, of course. The vast majority of 
federal funds are disbursed following a set formula, calculated by input metrics like the 
number of students in a district or state. For example, Title I of the Elementary and Secondary 
Education Act allocates monies to schools based on how many eligible children they enroll — 
not the characteristics of those students, nor how well (or poorly) those schools are doing at 
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"I believe it is critical for the 
federal government and the 
private sector to collaborate 
to identify what works and 
to bring the substantial 
financial resources required 
to the table." -Charles Harris, 
SeaChange Capital Partners 
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educating those students or otherwise performing. However, the Department also awards 
competitive grants; depending on the program, these funds may be awarded to agencies like 
districts and states, but also to outside organizations and individuals. But competitive grants 

are still a relatively small percentage: in fiscal year 2010, just 
$7.2 billion of the $64.1 billion in discretionary federal 
education spending was allocated through competitions 19 ; by 
comparison, more than $43 billion was allocated through 
three of the largest formula-based programs: Title I under 
ESEA, the Individuals with Disabilities Education Act (IDEA) 
grants to states, and the Pell Grant program for college 
students. 20 Roughly $2 billion of these discretionary programs 
sit within the relatively new Office of Innovation and 
Improvement (Oil), which was created in 2002 and managed 
the i3 grants program. With the combination of vast formula- 
driven funds and narrowly focused competitive grants, these 
streams of federal money are rarely coordinated in ways that 
encourage the development of brand-new innovations or the 
scale-up of successful solutions. Moreover, as entrepreneur Larry Berger and scholar Patrick 
McGuinn recently pointed out, “While federal spending on innovation is a vanishingly small 
part of the overall education pie, such funding is almost non-existent at the state and district 
level, so federal dollars have outsized influence.” 21 



"While federal spending on 
innovation is a vanishingly 
small part of the overall 
education pie, such funding 
is almost non-existent at the 
state and district level, so 
federal dollars have outsized 
influence." -Larry Berger, 
Wireless Generation and Patrick 
McGuinn, Drew University 



Education and the Economic Stimulus 

As the 2008 presidential election approached, ESEA reauthorization moved to the back 
burner, awaiting a new presidential administration and Congress to determine its future 
course. The economic crisis also came home to roost in education as in other parts of the 
economy, with school systems making massive job cuts and sharply limiting spending. With 
its emphasis on creative problem-solving and its promise of greater productivity — better 
outcomes for less money — “innovation” became a buzzword on both sides of the aisle 
in education and beyond. By late 2008, it was becoming clear that it was time for the 
government to take drastic action to address the economy, and by early 2009, Congress passed 
the American Recovery and Reinvestment Act — commonly known as the economic stimulus 
package, which promised to put $787 billion to work in reviving the economy by saving jobs, 
creating new infrastructure and spurring technological advances as a means to long-term 
economic growth. 22 
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The $100 billion education component of the stimulus package included about $54 billion 
in the form of the State Fiscal Stabilization Fund to be delivered to state governors, who 
would distribute funds to local education agencies and institutions of higher education. While 
more than $48 billion of that fund was awarded directly to states on a traditional formula 
basis to help prevent job cuts in early learning, K-12 and higher education institutions, the 
remaining $5 billion was carved out to advance reform. 23 With so much pressure to support 
the status quo in education, setting aside this sum to be awarded competitively was no small 
feat. President Barack Obama had appointed as his Education Secretary Arne Duncan, 
who brought a reform-oriented approach to the role of the federal government. As CEO of 
Chicago Public Schools, Duncan had embraced entrepreneurial reforms like charter schools 
as mechanisms for improving student achievement and sparking wider change in the city’s 
schools. “Too often our department has focused on bureaucratic compliance and audits, not 
on accelerating student achievement,” he noted in a 2009 speech to grantmakers. “I want 
our department to become an engine of innovation, not a compliance machine. I want the 
department to provide powerful incentives to states, districts, and nonprofits to innovate — but 
at the same time leave most of the creative thinking and entrepreneurship for achieving our 
common goals in local hands.” 24 

Of the $5 billion set aside for reform under the stimulus, the competitive state grant program 
known as “Race to the Top” made up $4.35 billion and would be awarded to states that 
demonstrated they could make advances against the four education policy “assurances” that 
the stabilization fund specified: progress toward rigorous college- and career-ready standards 
and assessments, pre-K to college and career data systems, teacher effectiveness and equitable 
distribution, and interventions for the lowest-performing schools. By requiring certain 
reforms to be in place to qualify states for Race to the Top grants, the Department established 
incentives for states to make significant changes to policies that would advance performance- 
driven practices and innovation, including the development of common state standards and 
corresponding assessments. 

The remaining $650 million was set aside into another fund that districts and nonprofit 
groups “with a strong track record of results” could apply for — the program that would 
eventually be called Investing in Innovation. 



"The Invest in What Works and Innovation Fund" 

The original stimulus legislation made reference simply to an “Innovation Fund” that would 
be a part of the education package. Flowever, initial speeches and regulations in early 2009 
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began to hint at the shape the program would take, focused less on early-stage innovation than 
on taking successful practices to scale, as embodied in its original name: the “Invest in What 
Works and Innovation Fund.” 25 But by the time the program had made its way through the 
congressional policy-making process and was formally unveiled in August 2009, the unwieldy 
but precise name had been shortened to the “Investing in Innovation” (or “i3”) Fund, which 
was easier to remember and pronounce but set up a misperception in the field that the fund 
would focus only on the popular connotation of innovation as purely new and different. In 
fact, Secretary Duncan went to great lengths to try to clarify this in his speeches unveiling the 
program. “Educational innovation should not be confused with just generating more great 
ideas or unique inventions,” noted Secretary Duncan. “Instead we need new solutions that 
improve outcomes — and that can, and will, be used to serve hundreds of thousands of teachers 
and millions of students.” 

Unlike many of the Department’s discretionary grant programs, which are narrowly 
proscribed and result in a relatively small number of applicants, the i3 program was designed 
to increase and spread education innovation in several specific ways: 

» To increase the “supply” of innovative practices and programs in a way that was aligned 
with “demand” from schools, districts and states attempting performance-based reforms 
inspired by and/or supported by federal Race to the Top funds; 

» To shift the Department’s allocation of resources toward evidence anchored in student 
achievement outcomes; 

» To encourage scale based on those outcomes, by establishing a graduated evidence 
framework that would provide a rigorous bar for demonstrated proof of successful 
student-level progress — and by providing more funds for more evidence; and 

» To mobilize and align private sector resources around key priorities defined by the 

Department, and a common definition of evidence, so that supported innovations would 
be stronger and more sustainable. 

As explained earlier, the full cycle of innovation does include both early-stage ideas and the 
growth of proven approaches and tools. However, although combining the two in a single 
ambitious grant program may have been politically necessary, doing so with a name change 
that emphasized only innovation set up unrealistic expectations in the field with respect to 
the novelty of what would be funded. In our survey of funders and i3 applicants, respondents 
acknowledged the i3’s intent to do both, but funders chose “increasing scale by investing in 
what works” as the most important goal of i3, whereas applicants said “increasing innovation 
activity. ” As we’ll explain further below, the path forward may be to address different stages 
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of innovation in different ways. As a respondent to a survey conducted by Whiteboard 
Advisors put it, “The real issue is not whether to fund one or the other. The real issue is 
that there are no mechanisms to determine how to price and distribute capital to these very 
different sorts of areas.” 26 



Rolling Out the Investing in Innovation Program 

The announcement of the first round of the Investing in Innovation program unleashed a 
great deal of pent-up excitement and interest in educational innovation. “When I first heard 
about i3, 1 thought it could make a big difference in the way schools work together and get 
supported in different ways,” noted one of our interviewees, who later applied for an i3 grant. 
“The problems we are chasing down are so massive that if government money isn’t involved, 
there’s no chance of success,” one grantmaker told us. During the 30-day public comment 
period following the original Notice of Proposed Priorities on Oct. 9, 2009, more than 300 
letters and 1,000 comments were submitted by districts, states, entrepreneurs, grantmakers 
and interest group associations. 



Who Could Apply: Eligibility and Partnerships 

On March 12, 2010, final regulations were released that nailed down the shape of the initial 
program. It would “provide competitive grants to applicants with a record of improving 
student achievement, attainment or retention in order to expand the implementation of, and 
investment in, innovative practices that are demonstrated to have an impact” on student 
achievement and attainment. The original Notice of Proposed Priorities in October 2009 
actually sought to standardize this track record by requiring applicants to demonstrate that 
they had met or exceeded their state’s No Child Left Behind (NCLB) progress objectives for 
two or more consecutive years, or at least increased student achievement for the subgroups 
of students targeted under NCLB. This was one of the biggest bones of contention during the 
public comment period. “Many school districts objected to the requirement in the proposed 
rules that applicants show strong evidence of past success in order to justify funding for an 
innovative strategy, while many education researchers thought the department should be even 
stricter,” noted Education Week at the time. Siding with the prospective applicants in the field, 
the final regulations dropped this NCLB-based requirement. 

The final regulations also spelled out clearly the kinds of organizations that were eligible 
to apply: local education agencies (LEAs) — typically districts, although some individual 
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Timeline 



January 20 

President Barack Obama 
assumes office 



January 21 

U.S. Education Secretary 
Arne Duncan assumes office 



March 

Name changes from "What 
Works and Innovation 
Fund Recovery Plan" to 
"Invest in What Works and 
Innovation Fund" 



2009 



February 13 

February 17 Congress passes the 

ARRA takes American Recovery 

effect and Reinvestment Act 

of 2009 (ARRA) 

April 

Jim Shelton appointed 
Assistant Deputy Secretary 
of the Office of Innovation 
and Improvement at the 
U.S. Department of 
Education 



October 9 

Release of Notice of 
Proposed Priorities 
for Investing in 
Innovation (i3) fund 



November 9 

Public comments due 



■■ 2010 



February 4 

Call for peer 
reviewers posted 

April 1 

Intent to apply due 
Peer reviewer application due 



June-July 

Peer reviewers selected 
and trained; reviews of 
all applications 



SEPTEMBER 8 

Deadline for 
securing 20% private 
sector match 



SEPTEMBER 30 

Government 
obligated all i3 
funding 



MARCH 12 

Final application available 



MAY 12 

Grant application due 



AUGUST 5 

Highest rated applicants announced 
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schools and public charter school organizations are structured in this way — and nonprofits 
(including colleges or universities) partnering with one or more LEAs or a consortium of 
schools. This was unlike the “Growing What Works” legislation proposed back in 2007 that 
would have channeled funding toward LEAs but also a wider range of other organizations. 
The congressional statute for i3 explicitly excluded for-profit organizations from participating 
as applicants or official partners, and barred them from receiving subgrants outside of the 
traditional cumbersome procurement process that anyone receiving federal funding must 
adhere to. Some of our interviewees felt that the Department interpreted the legislation 
conservatively and could have specified wider latitude on how subgrants worked in order 
to better engage private-sector companies, but they offered no specific examples for how to 
accomplish this or how other agencies have done so. The field of prospective applicants was 
narrowed further by requiring nonprofit organizations either to name the specific partner 
LEAs they would work with, or at least describe the characteristics of such partners and the 
process that would be used to select them. Similarly, LEA applicants had to demonstrate 
partnerships with the private sector, including commitments to match 20 percent of the 
requested federal funding with private funds (more on this later). 

Together, these eligibility requirements foreshadowed a pool of applicants and grantees 
made up of existing organizations that had already addressed K-12 schooling in some way, 
and limited to those who were willing and able to partner with LEAs or specific schools. By 
barring for-profit organizations from participating, it was virtually certain that the applicant 
pool — and the eventual grantee list — would miss out on some of the most promising 
technology advances, where the predominant support has come from private investors and 
most entrepreneurs have chosen to create for-profit businesses to attract engineering talent. 

“I think it is an incomplete attempt to spur innovation. There’s no reason to assume that K-12 
or higher education or nonprofits have a lock on innovation,” said one nonprofit leader whose 
organization later won an i3 Development grant. “Why not get the best of the best to push 
the envelope?” 

Requiring the organizations either to be a local education agency (LEA) or to have already 
demonstrated success in working with LEAs or schools emphasized respect for a track record 
in the field, but it also meant that few truly new organizations were eligible to apply — and 
those that consisted of newly formed teams were penalized in the application scoring for that 
newness, which may also have inhibited new efforts and contributed to the sense expressed by 
some that it was largely “the usual suspects” who participated. “A partnership that includes 
a nonprofit organization must demonstrate that the nonprofit organization has a record 
of significantly improving student achievement, attainment, or retention through its work 
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with an LEA or schools,” noted the Department’s Frequently Asked Questions document, 
which was updated no less than five times throughout the application process. “Although 
a partnership is not prohibited from including a ‘new’ nonprofit organization as an official 
partner, such a nonprofit organization would be unlikely to have such a record.” 27 While this 
language referred to the lead partners only and those applicants were free to engage any other 
partners — including those that might be newer or lack a K-12 education track record of any 
kind — the Department’s communication about this issue and the complication of interpreting 
such subtleties led most applicants to shy away from pulling any new organizations into their 
efforts lest it jeopardize their chances of winning a grant. 

Also missing from the equation were nonprofits that couldn’t (or didn’t want to) partner 
with a local education agency — some because of their structural focus on learning outside 
of school, others because of a belief that they could have a bigger or better impact working 
independently. And many small organizations and LEAs, especially rural districts, simply 
didn’t have the capacity — time or people, or money to buy either one — to take on such an 
extensive application. (Although Department officials point out that applicants with limited 
access to resources could request a waiver of some or all of their match, no one requested such 
a waiver. ) 



What Applicants Had To Address: Absolute Priorities, Competitive Preferences and the 
Evidence 

The original i3 legislation approved by Congress allowed the Education Secretary wide 
latitude to define in the regulations what applicants had to focus on and what their 
applications would need to include. First and foremost, the Department required all 
applications to focus on high-need students and the extent to which the LEA or partnership 
had significantly closed achievement gaps or increased student achievement, graduation rates, 
teacher or principal quality or other factors among those groups. Each application also had 
to focus on one of four “absolute” priorities, aligned with the four “assurances” of the State 
Fiscal Stabilization Fund part of the stimulus legislation (and thus also aligned with the related 
Race to the Top state competitive grants process): improvements in teacher effectiveness, 
enhanced data systems, college- and career-ready standards and rigorous assessments, and 
improving achievement in low-performing schools. 28 

By orienting its priority areas around the same “assurances” that drove the Race to the Top 
program through which a large infusion of state spending would be funneled, i3 sought to 
align the “supply” of new solutions with the “demand” being generated by the 12 states 
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that won RTT funds — not to mention the many other states that pursued related reforms 
in the hopes of winning RTT funds. These four “absolute” priorities also opened the door 
for a diverse range of organizations and initiatives to apply for funding, appropriate to the 
“field scan” structure. Many applicants appreciated this approach, and saw it as a way for 
the federal government to actively empower leaders in the field as legitimate sources for 
innovation. But applicants were required to choose just one absolute priority, which may have 
had the effect of favoring narrower solutions rather than more comprehensive or integrated 
innovations. 




Absolute Priorities and Competitive Preferences in i3, 2010 







Teacher and Principal 
Effectiveness 




Early Learning 
(0 or 1 point) 






Enhanced Data Systems 




College Access and 
Success (0 or 1 point) 


for high needs 
students 


+ 


College- and Career- 
ready Standards and 
Assesments 


+ 


Serving Students with 
Disabilities and Limited 
English Proficient 
Students (0 or 1 point) 






Improving Achievement 
in Persistently Low- 
performing schools 




Serving Students in 
Rural LEAs 
(0, 1 or 2 points) 


Required for all applications 




Must address one 
Absolute Priority 




May address one or more 
Competitive Preference 



Source: U.S. Department of Education 
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Applicants could also earn a few bonus points on their applications for addressing as many 
as four “competitive” priorities: improve early learning outcomes, support college access and 
success, address the unique needs of students with disabilities or limited English proficient 
students, or serve schools in rural LEAs. 

By far the most significant innovation that i3 brought to the table was its evidence framework 
and the related establishment of stages, which acknowledged that innovation progresses 
through a series of stages of maturity, from novel to promising to proven to replicable. 
Though controversial, this evidence framework established a rigorous level of evidence 
across all stages, even the very earliest; dictated that different amounts and types of evidence 




Types of Awards Available Under i3 



Development 




Estimated 

Funding Up to $5MM/award Up to $30MM/award Up to $50MM/award 

Available 

Evidence Reasonable - research 

Required findings or hypotheses, 

including related research 
or theories in education 
and other sectors 

Scaling Able to further develop Able to be scaled to the Able to be scaled to the 

Required and scale regional or state level national, regional, or 

state level 



Moderate - either high Strong - both high 

internal validity and internal validity and high 

medium external validity, external validity 
or vice versa 



Source: U.S. Department of Education 
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would be required at each stage of an innovation’s development; and drove more funding 
toward innovations with more evidence. This was a novel approach for a Department whose 
programs often specify one narrow range for “winning” applications rather than a full cycle 
of progress. “Development Grants” would be awarded to proposals with “reasonable” 
levels of evidence of “research-based findings or hypotheses,” with a maximum award size 
of $5 million, and with the explicit expectation that they would plan to take the program 
to some greater scale. “Validation Grants” were established for proposals with “moderate” 
evidence, including at least one experimental or quasi-experimental study that warranted 
further research in other contexts, with awards up to $30 million and an expectation that 
the project would scale to the regional or state level. The Department estimated that it 
would make up to 100 awards in each of these lower-level categories. Finally, the “Scale- 
Up Grants” category was reserved for just five awards, and would grant up to $50 million 
apiece to applicants that demonstrated “strong” evidence, including multiple experimental or 
quasi-experimental studies but also at least one “large, well-designed and well-implemented 
randomized controlled, multisite trial.” These Scale-Up grants were expected to scale up to 
the national, regional or state level. Across the three levels, there was no requirement for the 
specific number of students, schools or locales that needed to be reached by applicants, but the 
application did require “an estimate of the costs for the eligible applicant or others (including 
other partners) to reach 100,000, 250,000, and 500,000 students” — a level of specificity that 
confused many applicants and led them to pitch their scale in one of those directions. 

Applications would be screened and scored by peer reviewers, rather than department staff, 
based on a set of seven selection criteria that were consistent across all applications (see 
Figure 4 on page 23). The regulations specified different points available for each criterion 
in each evidence level — requiring evidence at all levels, even the earliest, though with a nod 
to the varying importance of different types of evidence and data based on the stage of an 
organization or program. “In the largest, scale-up grants, evidence is what matters most: 

It’s worth 20 percent of an applicant’s grade,” noted Education Week. “For the smallest, 
development grants, evidence is worth just 10 percent. But those smallest grants place a 
significant amount of weight on the need for the project, and the applicant’s track record — 
each is worth 25 percent of the final grade.” 29 The chart that follows shows the number of 
points available for each category across each of the evidence levels, beginning with the highest 
bar. For Scale-Up grants, strength of evidence (B) counted most strongly. For Development 
grants, the need for the project (A) mattered more, as did the applicant’s experience (C). While 
this made some sense given that these applicants would have little evidence of effectiveness, it 
meant that funding geared toward new ideas would go toward legacy players, rather than the 
kinds of new organizations that tend to come up with breakthrough concepts and products. 
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FIGURE 4 




Points Available for i3 Applications, 2010 



Selection Criteria 


Scale-Up Grants 


Validation Grants 


Development 

Grants 


A. Need for the Project and Quality of the 
Project Design 


15 


20 


25 


B. Strength of Research, Significance of 
Effect and Magnitude of Effect 


20 


15 


10 


C. Experience of the Eligible Applicant 


15 


20 


25 


D. Quality of the Project Evaluation 


15 


15 


15 


E. Strategy and Capacity to Bring to Scale 
(Scale-Up and Validation grants); Strategy 
and Capacity to Further Develop and Bring 
to Scale (Development grants) 


15 


10 


5 


F. Sustainability 


10 


10 


10 


G. Quality of the Management Plan and 
Personnel 


10 


10 


10 


Total Points 


100 


100 


100 



Source: U.S. Department of Education 



In our interviews and analysis, we found almost universal appreciation for the fact that i3 
emphasized evidence and acknowledged the different stages of an innovation’s development. 
Compared with other discretionary grant programs in education, applicants and funders 
agreed that i3 took a giant leap forward by setting forth a rigorous framework of evidence 
rooted not in input metrics like size of staff or hours of work but rather in outcomes metrics 
like the quality and scale of student achievement. “Steering money to effective programs is 
loads better than mailing ten billion bucks to districts so they can keep doing the same old 
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thing,” i3 skeptic Rick Hess of the American Enterprise Institute grudgingly conceded. 30 
“Unlike most federal grant programs, this program required an evidence base — you needed to 

show that your program actually impacts achievement for 
kids,” one applicant, whose organization ultimately won a 
Development grant, told us. “There need to be more grants 
programs that give money based on how students are 
achieving versus based on demographics or location. ” 

In general, the overwhelming focus on evidence in the 
application — regardless of stage — forced applicants to 
consider deeply the impact they sought and the role of 
evaluation in their work. Done well, evaluation is time- 
consuming and expensive, and the ability to use philanthropic 
or government resources to support this work is relatively rare. “i3 legitimized the importance 
of third-party research,” noted one winning Scale-Up applicant. “We didn’t have to spend 
it out of our own money and the results will probably play a huge part in our fundraising 
goal,” agreed another winning Scale-Up applicant. “We were grumpy about the evidence base 
initially,” one Validation grantee told us. “What it did do was motivate us to go get some 
quality research — it cost us some money but we had the resources to do so. ” 

However, there were mixed opinions about the appropriate evidentiary requirements at 
each level. At the Development stage, there was a sense that the selection criteria might have 
encouraged more break-the-mold innovations and organizations if the applicant’s experience, 
capacity to scale, and quality of the management team were assessed in other ways. “Neither 
the iPhone or iPad teams at Apple would have been able to meet this standard to get the 
funds to initiate these projects,” noted one Development applicant who did not win a grant. 
“It would have been helpful for the USDOE to better define what it means by innovation 
and how much risk it is willing to bear.” The applicant suggested that the USDOE replace 
questions about the applicant’s experience with ways to assess the organization or team’s 
past innovations and their capacity to manage risk, and noted that individuals’ newness 
to the organization should not be mistaken for lack of experience. “When an organization 
embarks on a new project, it is typical to hire talented individuals with expertise in the area. 
Organizations should always try to apply the best talent to the goal at hand,” the applicant 
added. 

At the Validation and Scale-Up stages, many felt that, prior to i3, very few organizations — 
nonprofits and LEAs alike — had the necessary financial resources to invest in the expensive 



"Steering money to effective 
programs is loads better than 
mailing ten billion bucks to 
districts so they can keep 
doing the same old thing." 
-Rick Hess, American 
Enterprise Institute 
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experimental or quasi-experimental studies required to meet these criteria. This rigid evidence 
definition may have skewed the applicant pool toward more of the “usual suspects.” “We 
weren’t able to have the gold standard of evidence because we didn’t have the foresight 
to randomly assign students or study lottery grantees versus losers,” notes one winning 
Development applicant. “They defined success in a way that was really difficult to accomplish. 
They needed to allow for more flexibility for metrics and let organizations develop the best 
ways to show the results and how they make them work.” The rigorous evidence requirements 
(especially applied through a rigid peer review process that seems to have drawn more- 
traditional “peers,” as we’ll explain further below) also favored those whose work could 
be easily and definitively measured, rather than new, comprehensive or multidimensional 
interventions. “I think there certainly should be high standards in doing this work, [but] I 
think we have to have a mixed set of evidence standards rather than just a randomized field 
trial to see what is working,” noted one Development applicant who did not win a grant. 



The Application Process 

The Department released regulations on March 12, 2010, and strongly encouraged 
applicants to send an email indicating their intent to apply by April 1 — naming the applicant, 
the evidence level the project would apply under, and the absolute priority — so that the 
Department could get a sense of how it would have to structure its decision-making and 
peer review processes. More than 2,500 organizations filed an intent to apply 31 , with final 
applications due just six weeks later, on May 12. 

Applicants worked quickly, and many of those we interviewed appreciated this forcing 
function and the opportunity to focus on some of the things they had wanted to do but hadn’t 
yet prioritized due to a lack of funding. In a matter of weeks, applicants pulled together 
partnerships and consortia that would otherwise have taken months or years. “Half of the 
value of i3 is just getting ideas on paper and forcing the execution,” said one Scale-Up grantee. 
However, other organizations (mostly those at the later Validation or Scale-Up stage) were 
adamant that the i3 process was only valuable because its field-scan approach allowed enough 
latitude to complement and align with their existing plans or sustainability goals. “We couldn’t 
bank on the scoring process and didn’t want the grant to pull us away from our strategy,” said 
one Validation grantee. “We saw this as an opportunity to build stability in our contracts, as a 
way to build five-year relationships with districts with initial dollars coming from the federal 
government and then having the districts assume more costs.” 
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The requirements that applicants partner with one another — mandating that LEAs partner 
with the private sector for their matching funds, and that nonprofits apply in partnership with 
schools and school systems — fostered a collaborative approach that some of the applicants 
we spoke with appreciated as a learning opportunity. One tool touted as a solution for this 
need was the Open Innovation Portal, which was launched in February 2010 as a public- 
private partnership itself (operated by the Department, the White House Office of Science 
and Technology Policy and STCIWenCorps.org). It was open to applicants, partners and 
potential funders, and was described as “a Web 2.0 innovation ecosystem” that would create 
a collaborative community around ideas and also a marketplace “that taps the ‘wisdom 
of the community’ to identify and resource the most promising ideas in education.” While 
none of our interviewees cited the Open Innovation Portal as a resource for finding money 
or partners in the i3 process, the Department says more than 5,000 members have joined the 
Portal, creating more than 1,000 connections and posting more than 250 ideas to improve 
education. 32 The Department also asked applicants to “participate in, organize, or facilitate 
‘communities of practice,’ which are defined as a group of grantees that agree to interact 
regularly to solve a persistent problem or improve practice in an important area.” In addition 
to at least annual meetings for all project directors, the Department is also considering 
providing several tools to support grantee communities of practice around thematic areas and 
grantee challenges. But the value of this collaborative effect will be unclear for quite some 
time, and if history is any indicator, assessing the impact of this knowledge development part 
of the program is likely to take a backseat to the evaluation of individual grantees’ activity. 

The Department made many elements of the process very transparent relative to other 
competitive grant programs, which one Validation grantee called a “breath of fresh air.” 
Generally one critique of federal grants processes is that officials publish only application 
guidelines and criteria, and then little to nothing is shared before they finally announce award 
grantees. In this instance, the Department of Education published the notice of proposed 
priorities to obtain public comment (which is not usually necessary for the first competition 
under a new program) months in advance of issuing final regulations. The Department 
also hosted three application workshops with more than 1,000 attendees (and another 
2,000 joining by webinar), posted materials from those sessions online, regularly updated 
an expansive list of frequently asked questions (FAQs), and posted interim lists of all those 
organizations that filed an intent to apply, along with detailed information about the winning 
applications and other highly rated applications that did not make the final cut. However, 
this transparency may be an example of the theory of rising expectations: even though this 
program was vastly more transparent than most in the federal government’s history, releasing 
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so much information may have paradoxically spurred a demand for even more detailed 
information, including how eligibility was determined and how raw reviewer scores were 
standardized into scale scores (more on the latter in the next section). The Department also 
estimated that it would award up to five Scale-Up awards and up to 100 Validation awards 
and 100 Development awards, unintentionally setting the unrealistic expectation in the field 
that hundreds of organizations would benefit — though just a few dozen would ultimately win. 

By the time mid-May rolled around, 1,698 applicants chose to throw their hats into the 
ring — a remarkable number, but less than 70 percent of the more than 2,500 that had 
indicated their intent to apply six weeks earlier. As prospective applicants mulled their options, 
the Scale-Up level surely proved the most daunting: although 87 indicated an intent to apply at 
this evidence level, just 22 percent of them (19 applications) were ultimately submitted, while 
more than 65 percent of the expected number of applications at the Validation level and close 
to 80 percent of those at the Development level made their way in. 



FIGURE 5 



Intended and Actual i3 Applications, 2010 



Grant Type 


Intent to 
Apply 


Applications 

Received 


Scale-Up 


87 


19 


Validation 


527 


355 


Development 


1666 


1324 


NA/Unclear 


165 


0 


TOTAL 


2445 


1698 



Source: U.S. Department of Education 
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Decision-Making Process: Peer Review and Application Scoring 

Recognizing the difficulty of identifying effective innovations — particularly at the early 
stages — the Department initially had hoped to engage intermediaries in the process, but 
congressional statute did not allow it. As a result, the Department had to handle selection in- 
house, and ended up engaging peer reviewers to score the massive stack of applications. While 
the use of peer reviewers is not unusual for a government grant program, many questioned 
the decision to rely solely on the scores of a large pool of peer reviewers to identify quickly — 
and on paper alone — which innovative ideas were worthy of support and which established 
programs demonstrated sufficient evidence to merit enormous sums of money to help them 
grow. As a comparison point, other federal programs designed to award grants for innovative 
programs and practices, such as the National Science Foundation and the National Institutes 
of Health, rarely put all the power in the hands of peer reviewers. More often, peer reviewers 
are a check on the government’s process and a way to supplement government knowledge with 
industry expertise. A number of interviewees hypothesized that perhaps the recent Reading 
First controversy led Department officials to err on the side of caution by relying exclusively 
on peer review judgment and leaving aside their staff’s own judgment about the peer review 
process and the resulting grantees. 

The process of choosing those peer reviewers was itself daunting: more than 1,400 peer review 
applications were submitted, with 346 reviewers ultimately chosen to sift through and score 
the nearly 1,700 applications. Some reviewers were chosen for their subject matter expertise 
and others (with selection help from the Institute of Education Sciences) for their knowledge 
of research and evaluation. 33 The need for so many reviewers made it challenging to find 
qualified reviewers who had no conflicts of interest with applying organizations. Stringent 
criteria designed to eliminate conflicts of interest meant that many knowledgeable people were 
disqualified: not only employees and consultants hired by i3 applicants were off limits (not to 
mention anyone else with a personal financial stake in the outcome), so were reviewers with 
what was termed an “indirect conflict of interest” who had provided any kind of advice or 
assistance to any of the i3 applicant projects. 34 Many of the most knowledgeable people were 
taken off the table — or, if allowed to review, were assigned to applications from different 
content areas and different states, and were unable to chair a review panel. 35 Who was left? 
“District data officers and retired professors,” observed one reviewer. “After working in this 
field for eight years, I only recognized five names on the list of 346 peer reviewers,” another 
innovation observer told us. While expertise is important for peer reviewers, another observer 
noted that those with academic credentials in education per se are often more interested in 
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content and curriculum, rather than systems change, and may have biased the peer review 
process in favor of more incremental innovations. 

Once applications were submitted by May 12, the process of screening and scoring 
applications kicked into high gear, where it remained throughout the summer. Those who 
were selected as reviewers were organized into “panels,” or groups of reviewers, according 
to the absolute priority their applications addressed. Each panel was made up of three to five 
peer reviewers who read the same set of 10 to 20 applications. For Scale-Up and Validation, 
the panel included three who looked at subject matter and two who rated based on research/ 
evaluation. Development grant applications took a slightly different path, with all applications 
scored first by subject-matter reviewers, and then the top 100 applications advancing to a 
second tier, where they were then reviewed by two research/evaluation experts. 36 Reviewers 
were given explicit instructions to review and score each application separately, and not 
compare similar applications directly with one another. 

Given the tight timeline and sheer volume of applications, reviewers received limited 
training: one orientation conference call to help them navigate the 10 to 20 applications 
they and their fellow panelists would be screening 37 , and several additional calls during the 
process to “norm” their scores with those of other reviewers who had scored those same 
applications. This may have had the effect of watering down the reviewers’ assessments: 
one peer reviewer we spoke with felt that when a small group of reviewers brought differing 
perspectives to the evidence, they found themselves “regressing to the mean” and deferring 
to the most conservative reviewer. The Department took some measures to try to reconcile 
the tension between detailed scoring specifications intended to protect against bias or fraud, 
and the inevitable array of skills and judgments across a large set of reviewers. To adjust for 
reviewer bias, the Department converted raw scores into scaled scores for the Development 
and Validation categories — a calculation whose specifics weren’t publicized, but which 
raised questions for some. “In large-scale competitions, the Department uses a statistical 
standardization process which adjusts for the effect of any large-scale differences in reviewer 
approaches to assigning raw scores,” noted the Department’s i3 FAQ. “However, the smaller 
number of applications in the Scale-Up category did not support the use of standardization, 
and therefore raw scores were used for these applications.” 

Applicants who later read reviewers’ comments noted that reviewers did not seem to have 
a strong grasp on the content areas they scored, nor on the practice of effective evaluation. 

“It was obvious that the peer reviewers were not all trained on understandings of what good 
evaluation is, nor were they well versed in the topic,” noted one Scale-Up applicant who did 
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not win a grant. Another Development applicant who did not win suggested that individual 
reviewers’ scores could have been weighted based on their content knowledge, with more 
knowledgeable peer reviewers’ comments counting for a greater share of the score. Although 
applicants who did not win may have been most critical of the peer review process, this 
criticism was fairly widespread. Among respondents to the Whiteboard Advisors survey, 65 
percent agreed that “The scoring was too random with insufficient protocols and controls for 
the reviewers.” Several of our interviewees thought the outcomes might have benefitted from 
throwing out the highest and lowest scores, as is done in some Olympic sports (where it is 
called “trimmed means”) to minimize the effects of extreme ratings by judges biased in one 
direction or the other. (Notably, although the second round of i3 will continue to rely upon 
peer review, reviewers will not assign scores based on the strength or weakness of the evidence 
itself; instead, actual evidence will be included as an appendix and reviewed by the Institute of 
Education Sciences. 38 ) 

A final criticism of the application scoring process was that it reduced programs and 
organizations to their paper applications, with no in-person diligence to meet teams, visit sites 
or review evidence deeply. “An application process like this is not a very good way to assess 
the full weight of the evidence of a project’s effectiveness, because applicants can choose what 
to highlight or exclude from their proposals,” noted Sara Mead of Bellwether Education 
Partners in her blog (where she also disclosed that she had helped write some i3 applications). 
“So as long as an applicant has at least some evidence that can be presented as meeting 
‘moderate evidence’ criteria, the points that applicants actually receive depend on savvy grant- 
writing as much as what the full weight of the evidence actually says.” 39 This was a sentiment 
repeated by a number of applicants as well — both grantees and those not selected. 

According to Department officials, they initially hoped to have a two-stage process for 
Development stage applicants — the ones with the least to show on paper about the results 
and promise of their work — that would have included an initial executive summary, followed 
by full proposals solicited from some, and then site visits and interviews with the finalists. 
However, the process of drafting, vetting and implementing i3 regulations — combined with 
ARRA requirements for getting money out the door by September 2010 — precluded such a 
staged application process. 



Selection: Announcement of Highest-Rated Applicants and Reactions From the Field 

The list of the 49 “highest-rated” projects — eligible for $646 million in grants once they had 
secured their 20 percent in matching funds — was released on Aug. 5, 2010. Nearly half of 
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the funds would go to 15 Validation grantees, with four Scale-Up applicants awarded $195 
million between them, and 30 Development grantees splitting $140 million. The grantees 
spanned 42 states and two territories (D.C. and American Samoa) and 250 project locations. 
California was home to the greatest number of i3 grantees, with eight, followed by five in both 
Massachusetts and New York, four in Washington, D.C., and three in Maryland. 




Highest Rated i3 Applications, 2010 



CATEGORIES AND APPLICANTS 


AMOUNT 


Scale-Up Grants (4) 


$194,878,659 


KIPP Foundation: 

Success as the Norm: Scaling-Up KIPP's Effective Leadership 
Development Model 


$50,000,000 


Ohio State University: 

Reading Recovery: Scaling Up What Works 


$45,593,170 


Success for All Foundation: 

Scale-Up and Evaluation of Success for All in Struggling Elementary 
Schools 


$49,285,513 


Teach For America: 

Scaling Teach for America: Growing the Talent Force to Ensure All Our 
Nation's Students Have Access to a Quality Education 


$50,000,000 


Validation Grants (15) 


$310,699,851 


ASSET Inc. (Achieving Student Success through Excellence in 
Teaching): 

ASSET Regional Professional Development Centers for Advancing 
STEM Education 


$ 22,308,433 


Children's Literacy Initiative: 

Children's Literacy Initiative's Model Classroom Innovation for Raising 
Teaching Quality and Increasing Student Literacy Achievement 


$21,726,296 
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CATEGORIES AND APPLICANTS 


AMOUNT 


Validation Grants (continued) 




Council for Opportunity in Education: 

Using Data to Inform College Access Programming in the 21st 
Century High School (Using DICAP) 


$ 20,264,447 


George Mason University: 

Virginia Initiative for Science Teaching and Achievement (VISTA) 


$ 28,455,346 


Johns Hopkins University - Center for Social Organization of 
Schools: 

Validating the Talent Development-Diplomas Now Secondary School 
Turnaround Model 


$ 30,000,000 


New Schools for New Orleans: 

Scaling the New Orleans Charter Restart Model 


$ 28,303,909 


Niswonger Foundation: 

Northeast Tennessee College and Career Ready Consortium 


$ 17,751,044 


Parents as Teachers National Center: 

Improving Educational Outcomes for American Indian Children 


$ 14,253,165 


President and Fellows of Harvard College - Graduate School of 
Education: 

Project READS: Using Data to Promote Summer Reading and Close 
the Achievement Gap for Low-SES Students in North Carolina 


$ 12,773,136 


School District No. 1 of the City and County of Denver, State of 
Colorado: 

Collaborative Strategic Reading Colorado (CSR-CO) 


$ 25,202,752 


Smithsonian Institution - National Science Resources Center, LASER: 

The LASER Model: A Systemic and Sustainable Approach for 
Achieving High Standards in Science Education 


$ 25,581,105 


The Curators of the University of Missouri - eMINTS National 
Center, Academic Affairs: 

eMINTS Validation Project 


$ 12,277,674 


The New Teacher Project, Inc.: 

Teacher Effectiveness and Certification Initiative (TEACh Initiative) 


$20,829,095 


Utah State University - Center for Persons with Disabilities: 

New Mexico K-3 Plus Extended School Year Validation Study 


$ 1 5,282,720 
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CATEGORIES AND APPLICANTS 


AMOUNT 


Validation Grants (continued) 




WestEd - Teacher Professional Development Program: 

Scaling Up Content-Area Academic Literacy in High School English 
Language Arts, Science and History Classes for High Needs Students 


$ 18,166,181 


Development Grants (30) 


$140,399,885 


Advancement Through Opportunity and Knowledge - Children 
Youth and Family Collaborative: 

District-wide program development, expansion and evaluation of 
the Education Pilot Project (EPP) for foster youth and preparation for 
statewide scale-up. 


$ 3,649,580 


Alliance for College-Ready Public Schools: 

CollegeYes 


$ 4,989,786 


American Federation of Teachers Educational Foundation - AFT 
Educational Issues: 

American Federation of Teachers Educational Foundation Educator 
Evaluation for Excellence in Teaching and Learning Consortium 


$5,000,000 


AppleTree Institute for Education Innovation: 

Every Child Ready 


$5,000,000 


Bay State Reading Institute: 

The Data Driven School Transformation Partnership. A project of 
the Bay State Reading Institute and 12 Massachusetts Elementary 
Schools. 


$ 4,997,492 


Beaverton School District 48J - Teaching and Learning: 

The Beaverton School District Arts for Learning Lessons Project 


$ 4,041 ,659 


Bellevue School District: 

Re-imagining Career and College Readiness: STEM, Rigor, and Equity 
in a Comprehensive High School 


$4,149,813 


Board of Education of the City of New York - Division of Talent, 
Labor and Innovation, Office of School of One: 

New York City Department of Education School of One 


$4,999,560 


Boston Plan for Excellence in the Public Schools Foundation: 

Boston Teacher Residency: Building the Pipeline of Effective Teachers 
for Turnaround Schools 


$4,855,617 
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CATEGORIES AND APPLICANTS 


AMOUNT 


Development Grants (continued) 




Boys & Girls Clubs of Greater Milwaukee: 

The Milwaukee Community Literacy Project 


$4,142,965 


California Education Round Table Intersegmental Coordinating 
Committee - Alliance for Regional Collaboration to Heighten 
Educational Success (ARCHES): 

STEM Learning Opportunities Providing Equity 


$ 4,982,527 


Corona-Norco Unified School District 

Curriculum and Instruction, Educational Services: Write Up 


$ 5,000,000 


District 75/New York City Department of Education: 

Everyday Arts for Special Education 


$ 4,633,397 


Education Connection - Center for 21st Century Skills, School 
Services: 

Science, Technology, Engineering, and Math Education for the 21st 
Century (STEM21) 


$ 4,473,481 


Erikson Institute: 

Achieving High Standards for Pre-K-Grade 3 Mathematics: A Whole 
Teacher Approach to Professional Development 


$ 4,999,993 


Exploratorium - Institute for Inquiry: 

Integrating English Language Development and Science: A 
Professional Development Approach 


$ 2,984,628 


Forsyth County Schools: 

EngageME P.L.E.A.S.E. 


$ 4,738,500 


IDEA Public Schools: 

Rio Grande Valley Center for Teaching and Leading Excellence 


$4,945,998 


Iredell-Statesville Schools: 

COMPASS: Collaborative Organizational Model to Promote Aligned 
Support Structures 


$ 4,999,036 


Jefferson County Board of Education DBA Jefferson County Public 
Schools - Jefferson County Public Schools, High Schools: 

Making Time for What Matters 


$ 4,999,458 





Bellwether Education Partners 35 



CATEGORIES AND APPLICANTS 


AMOUNT 


Development Grants (continued) 




Los Angeles Unified School District - Office of the Superintendent: 

L.A.'s Bold Competition - Turning Around and Operating Its Low- 
Performing Schools 


$ 4,880,392 


Montgomery County Public Schools - Office of Curriculum and 
Instructional Programs: 

North Star 


$ 4,999,634 


National Forum to Accelerate Middle-Grades Reform: 

Schools to Watch: School Transformation Network 


$ 4,999,969 


Plymouth Public Schools: 

New England Network for Personalization and Performance 
(NETWORK) 


$ 4,992,945 


Saint Vrain Valley School District - Priority Schools: 

St. Vrain Valley School District i3 Project 


$ 3,608,880 


School Board of Miami-Dade County, Florida - Intergovernmental 
Affairs, Grants Administration, and Community Services, Grants 
Administration: 

Florida Master Teacher Initiative 


$5,000,000 


Search Institute: 

The Building Assets-Reducing Risks Program: Replication and 
Expansion of an Effective Strategy to Turn Around Low Achieving 
Schools 


$4,999,711 


Take Stock in Children Inc.: 

Graduation and Higher Education for Tomorrow 


$ 4,999,947 


The Achievement Network LTD: 

Improving Data Use in Schools: Expanding the Achievement Network 
Model 


$ 4,999,987 


The Studio in a School Association, Inc.: 

Arts Achieve: Impacting Student Success in the Arts 


$ 4,372,801 


TOTAL GRANTS AWARDED 


$645,978,395 



Source: U.S. Department of Education 
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All absolute priority areas were addressed, and mostly evenly across the board, but at the 
Scale-Up stage, the four highest-rated applicants were divided across just two of the priority 
areas, teacher and principal effectiveness (Teach For America and KIPP Foundation) and 
school turnarounds (Success For All Foundation and Ohio State University’s Reading Recovery 
program). Largely because of the size of those four grants, those two absolute priorities 
gobbled up more than two-thirds of the total dollars allocated through i3. The standards and 
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i3 Awards by Absolute Priority and Stage, 2010 



Absolute Priority 


Development 


Validation 


Scale-Up 


TOTAL 


1 . Innovations that 
Support Effective 
Teachers and 
Principals 


6 

($27,418,676) 


4 

($96,213,489) 


2 

($100,000,000) 


12 

($223,632,165) 


2. Innovations that 
Improve the Use of 
Data 


7 

($33,385,066) 


2 

($33,037,583) 


0 

($0) 


9 

($66,422,649) 


3. Innovations that 
Complement the 
Implementation 
of High Standards 
and High-Quality 
Assessments 


10 

($47,002,639) 


5 

($96,084,437) 


0 

($0) 


15 

($143,087,076) 


4. Innovations 
that Turn Around 
Persistently Low- 
Performing Schools 


7 

($32,631 ,375) 


4 

($87,839,794) 


2 

($94,878,683) 


13 

($215,349,852) 


Total Grants 
(Funding) 


30 

($140,399,885) 


15 

($310,699,851) 


4 

($194,878,659) 


49 

($645,978,395) 



Source: U.S. Department of Education 
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assessments priority claimed the greatest share of grantees (33 percent) but just 20 percent 
of the funding, with data use innovations making up the final 20 percent of grantees and 10 
percent of the funding. 

When the highest-rated applicants were announced, the reactions were somewhat predictable 
within the context of the program design. Many were expecting more revolutionary, 
unexpected players and projects, given the name of the program, but the eligibility 
requirements barring for-profits and selection criteria favoring established organizations made 
that unlikely. “To some degree, the program was misnamed,” noted one Validation grantee. 
“From the name, people thought it was an opportunity to try new, untested ideas. That 
wasn’t actually the goal. The name drew great amounts of interest and lots of competition, 
but that wasn’t what you found in the requirements of the grant.” Moreover, the evidence 
framework emphasized experimental and randomized controlled studies, which are expensive 
to conduct, meaning only the most-well-heeled organizations had been able to conduct such 
research prior to the competition. While almost everyone we interviewed gave the Department 
credit for trying to establish rigorous evidence frameworks, many expressed concerns about 
the appropriateness of the exact standards for each level. On a survey of D.C. “Insiders” 
conducted by Whiteboard Advisors in October 2010, shortly after the grantees were 
announced, “the majority of Insiders (68%) believed these standards were not effective in 
terms of helping the Department identify and scale innovative practices and programs.” 

Finally, as noted above, the makeup of the peer review panels and the way they were asked to 
screen proposals may have foreordained a relatively “safe” list of final grantees. This was also 
a political reality, as pointed out by one Validation applicant who didn’t make the cut. “In a 
country that is only just beginning to understand what educational innovation looks like, it’s 
not entirely surprising that the Department would make bets on some strong, well-proven 
organizations and ideas, which have an excellent chance of showing the American public the 
good that this kind of work can do,” said Ted Mitchell of NewSchools Venture Fund. “Such 
successes may open space for a mix of grants that tilts further toward pure innovation in 
future rounds — but the Department, and those of us who support innovation, will need to 
make the case now in order for future rounds to even exist.” 40 

Mitchell was an outlier, though. When the grants were announced, dozens of reactions were 
posted online and featured in the media — mostly critical of either the lack of novelty of 
Development stage winners or of the game-changing prospects of Scale-Up grantees. “The 
essence of an innovation is the degree to which its novelty captures imaginations and the 
way its influence alters expectations as though mere word of its existence might provoke an 
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involuntary ‘Wow, that’s really cool!’ reaction,” wrote Steve Peha of Teaching That Makes 
Sense. “I am not hearing this involuntary reaction very much this week.” Investor Tom Vander 
Ark of Learn Capital (and a former foundation official) agreed, but gave the Department some 
credit for the later-stage grantees, noting, “The i3 program funded credible scaling efforts 
that will make incremental improvements to traditional schools — solid investments but not 
innovation.” A survey of D.C. “Insiders” conducted around the same time by Whiteboard 
Advisors found similar reaction. “In reading a majority of the i3 proposals, I am struck by the 
lack of innovative ideas that were submitted and ultimately funded,” noted one respondent. 
“Not enough real cutting edge stuff was funded. Almost nothing for technology and teacher 
effectiveness/quality,” noted another. 

At the Scale-Up level, some worried that funds were merely being allocated to continue 
growing organizations already on a path to do just that, rather than taking those programs 
to new places, solving new problems, or connecting with wider change. “I wouldn’t consider 
all those that won at scale to be able to fix the infrastructure, especially at a national level,” 
said one Development applicant who did not win a grant. Department staff point out that 
many of the larger grants contained less product innovation but more process innovation, 
with different ways of taking something that “works” to scale in a way that is sustainable 
and effective, such as Reading Recovery’s train-the-trainer model and KIPP’s development of 
school leaders. 

Given that much of the public criticism included minimal analysis of the winning programs’ 
actual goals and strategies, it is unclear if this skepticism is based on a thorough review of 
what the winners will do, or merely a review of names on a list. For example, many in the 
entrepreneurial sector expressed disappointment with some of the districts that won, but may 
not have understood that some of those districts’ projects echoed innovative entrepreneurial 
ideas. For instance, Department officials pointed out that in Georgia, Forsyth County’s project 
features personalized learning similar to the lauded School of One (also a highest-rated 
applicant), and that Los Angeles Unified School District is gearing its turnaround strategy 
toward a “portfolio school system,” an approach that many innovators have called for over 
the last decade to improve the management of large school districts. 

On the other hand, some experienced observers (whose expectations were perhaps a bit more 
sanguine) expressed appreciation for what had been accomplished. “This changed how we 
talk about education and made ‘innovation’ a center — which is different from just reform,” 
said one interviewee. “In many ways this was an unbelievable success,” another interviewee 
told us. “In a remarkable timeframe, they achieved significant things, including changing the 
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FIGURE 8 



i3 Awards by Competitive Preference and Stage, 2010 



Competitive Preference 


Development 


Validation 


Scale-Up 


TOTAL 


5. Innovations for Improving 
Early Learning Outcomes 


7 


3 


3 


13 


6. Innovations that Support 
College Access and Success 


14 


5 


1 


20 


7. Innovations to Address the 
Unique Needs of Students 
with Disabilities and Limited 
English Proficient Students 


19 


3 


4 


28 


8. Innovations that Serve 
Schools in Rural LEAs 


7 


8 


3 


18 



NOTE: Applicants could select more than one competitive preference, so totals add up to more 
than the number of winners. 



Source: U.S. Department of Education 



way the government operates and introducing a serious focus on evidence into the process, 
even though there were bumps and bruises along the way.” “It is true that some good projects 
were left out but any process is imperfect,” said one funder who has worked in the federal 
government. “It’s important to recognize that, do the best we can, and move on.” 

Some also expressed concern that there was insufficient attention to the pre-kindergarten, 
higher education and rural solutions that were submitted. As noted above, the Department 
awarded a small number of “competitive preference” points to applications that addressed 
these issues. “The i3 framework assigns very few points to these competitive priorities, with 
the effect that they serve only as tie-breakers for applications of equivalent quality under the 
selection criteria,” noted an Education Counsel analysis of the scoring system, although a look 
at the final scores shows that the difference between winning and losing did come down to just 
a few points in the Validation and Development categories. The most popular was competitive 
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preference 7, with more than half of the highest-rated applicants addressing the needs of 
students with disabilities or limited English proficient students — the latter of whom make up 
a significant proportion of the population in high-need communities where i3 applicants were 
concentrated, such as California and Texas. Among the highest-rated applicants, competitive 
preference 5 (Innovations for Improving Early Learning Outcomes) was the most rare: just 
14 percent of applicants noted it as competitive priority on their application, but a higher 
percentage (27 percent) of winners did. 

A number of our interviewees noted that although the competitive preferences channeled some 
funding toward areas that are often overlooked — particularly rural learning, early learning 
and college access — the way competitive preferences were structured and written proved to 
be largely ineffective for encouraging meaningful innovation in these areas. The competitive 
preference related to rural populations drew the most heated criticism. While the same 
proportion of winners addressed rural populations as picked the higher education preference, 
rural applicants felt that the deck was stacked against them. “The vagueness of the criteria 
and the extra value assigned to it encouraged many applicants with limited rural education 
experience to attach a small rural effort onto an otherwise urban program, ” claimed a Rural 
Trust report. Of the 19 highest-rated applicants who made the claim, “only two proposals 
are designed to operate entirely in rural schools. For most, the proportion of the total project 
effort that is rural-focused is small relative to the scale of the project, or too indeterminate 
to be estimated. In one instance there was actually no intent to engage in any rural school 
district.” 41 Department staff pointed out that while the actual number of rural i3 grantees 
was low, the per-capita dollars that will flow to those areas are relatively proportionate. 
(Nonetheless, this concern seems to have been heard loud and clear, given that the second 
round of i3 bumps “Improving Achievement in Rural LEA’s” up to an absolute priority.) 

The effect of winning or losing out on i3 funding has echoed beyond the direct flow of federal 
and private funds. Those who were chosen have enjoyed more positive attention in the press 
and from funders, they told us during our interviews. “i3 brought monetary value, but more 
importantly, it brought us visibility and affirmation of our work,” noted one Development 
grantee. Those not chosen as grantees worry about the stigma of not winning — particularly 
at the later stages where there were fewer applications — and about its impact on their ability 
to raise capital in the short term, while i3 looms large in the minds of funders. “It was a huge 
negative that we didn’t get money,” noted one Scale-Up applicant. “I think it might actually 
hurt us a little because people [that] gave to some other [similar] organizations might have 
given to us had we won.” In some of these cases, non-grantees are moving forward with 
the projects they had proposed (even, in at least one case, with the support of those funders 
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who had committed matching resources) but on a slower timeline; others reported dropping 
their projects entirely due to lack of funding (see below for more on the impact of i3 on 
philanthropic funding). 

Notably, in addition to differentiated evidence requirements and point allocations, the i3 
regulations acknowledged that different stages of programs and organizations need different 
types of support and accountability. Development and Validation awards were structured as 
traditional government grants while Scale-Up grantees entered into “cooperative agreements” 
with the Department. “That legal structure signifies that USED intends to be significantly 
involved in a partnership-type relationship with Scale-Up recipients in order to actively oversee 
and guide the projects, but will exercise the ordinary level of discretionary grant oversight 
for Validation and Development projects,” notes an Education Counsel analysis. 42 In both 
cases, reporting requirements are fairly standard: grantees must submit a final report after 
the project is complete, as well as quarterly reports required of all ARRA-funded projects 
that focus mostly on the use of funds and their relationship to saving or creating jobs (though 
the Department has added a question to this list about “the project’s progress in reducing 
inequities in the distribution of highly qualified teachers, implementing a longitudinal data 
system, and developing and implementing valid and reliable assessments for English language 
learners and students with disabilities”). However, as is typical of most Departmental funding 
and especially under the requirements of the stimulus, all funding was disbursed upfront, 
rather than in tranches as organizations meet specific progress milestones; the Department 
says it will monitor organizational spending over the course of the project, but funding isn’t 
explicitly tied to the accomplishment of intended outcomes, although future funding may 
be. Other government innovation investment efforts like the Defense Advanced Research 
Projects Agency take a more staged approach to funding, based on progress and milestones, 
particularly for early-stage innovations. 



The Public-Private Partnership: Matching Funds Requirement 

From the beginning, i3 was intended to be a public-private partnership. In the stimulus 
legislation establishing the i3 awards, one basis listed for the awards — right alongside “expand 
their work and serve as models for best practices” and “identify and document best practices 
that can be shared, and taken to scale based on demonstrated success” — was “to allow such 
eligible entities to work in partnership with the private sector and the philanthropic 
community.” 43 Over the past 50 years, such public-private partnerships have emerged as a way 
to address public or social needs through collaboration with the private sector, which can 
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include some combination of businesses, private investors, nonprofit organizations and 
philanthropies. The public sector is responsible for looking out for the common good, and 

brings deep and relatively predictable funds and operational 
capacity to the table, while the private sector tends to have 
access to a wider pool of talent and to operate in a more 
nimble way. The idea is that by leveraging the different 
capabilities and funding sources of the sectors, more can be 
accomplished — and in a more efficient way than either 
operating alone. However, public-private partnerships are also 
challenging because they require two (or three) very different 
sectors to work together across different constraints and 
operating norms. 

As detailed earlier, one element of the public-private partnership in i3 was the eligibility 
requirement that nonprofit applicants partner with LEAs or a consortium of schools. Whether 
a nonprofit or an LEA, each applicant had to demonstrate “that it has established one or more 
partnerships with the private sector, which may include philanthropic organizations, and that 
the private sector will provide matching funds in order to help bring results to scale. ” It was 
this latter piece — the requirement that all grantees match 20 percent of their i3 funding with 
private funds — that stands as one of the most talked-about aspects of the i3 fund. Financial 
support for grantees from the philanthropic community was intended to help provide 
a window into sustainability as well as a more nimble partner for ongoing support and 
accountability with grantees. The match could also serve as a “check” on the Department’s 
selection process: if a selected grantee couldn’t raise the required match, it might be a “false 
positive” that was not a viable innovation in the market (but, as some philanthropic critics 
have noted, there was no such mechanism for identifying “false negatives” that were not 
chosen for i3 grants but that philanthropists believed were more worthy of public and 
matching funds). 

On the whole, the philanthropic sector was very receptive to the federal government’s interest 
in applying innovation to improve public education. “With the downturn, foundations were 
retreating, and this was a good way to get them to increase spending to leverage government 
funds,” one funder explained. In spring 2009, national foundation leaders joining together 
as the Foundation Executive Group invited Arne Duncan to a meeting where he shared his 
emerging agenda and they gave him a paper summarizing their education priorities. In fall 
2009, Vartan Gregorian, president of the Carnegie Corporation of New York, organized a 
follow-up meeting with 10 foundation presidents that identified several overlapping priorities 
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between the foundations and the Department, including innovation in teaching and leadership 
and in new school models and designs. Those foundations agreed to work closely with 
one another and with Jim Shelton, Assistant Deputy Secretary at the Department and head 
of its Office of Innovation and Improvement, to coordinate philanthropic resources with 
the Department’s efforts. This group of foundation presidents and their staff became very 
engaged in ongoing conversations about how best to structure i3 as a successful public-private 
partnership. 

Early conversations between Department officials and the foundation community — including 
through the Foundation Executive Group — included several ideas for how to structure this 
partnership. The Department floated the idea of creating a pooled fund for the statutorily 
required matching funds, which did not sit well with foundations because they did not want to 
give up authority over the use of their individual funds. Still, in April 2010 — shortly before i3 
applications were due — 12 foundations did announce that they had committed $500 million in 
2010 funds “aimed at similarly aligned investments, making more than $1 billion available to 
help expand promising innovations in education.” 44 

Another area of discussion was the order in which a match would be required — before or 
after the government peer review process. In the initial proposed regulations, applicants would 
have been required to show evidence of their match in their application. Funders did not like 
this option, as it would have put them in the awkward position of prioritizing among their 
grantees — and doing so without knowing what the real full level of need would be across 
existing grantees and other interesting prospects. “ [The match] is unworkable in its demands 
of the philanthropic organizations,” noted Allan Golston, president of the Bill & Melinda 
Gates Foundation’s United States program, during the public comment period on proposed 
i3 regulations in fall 2009. “It makes philanthropies de facto gatekeepers for applicants by 
requiring the match at the time of application; it privileges those organizations that already 
have relationships with large foundations.” 45 In the final regulations, the match requirement 
was shifted to follow peer review, so that only the finalists would be required to secure 
matching funds. 

While this change met the needs expressed by funders during the public comment period, it 
left a few other funders displeased that they would have to follow the federal government’s 
decision-making process, rather than engaging with it in a true side-by-side partnership. 
Explicitly aligning their work with federal government priorities is relatively uncharted 
territory for philanthropists, especially individual donors and small local or regional 
foundations, but even for the big national foundations. Most foundations pride themselves on 
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their own distinct theories of change and giving strategies, and prefer to put money to work at 
their own pace. The structure of i3 flipped this on its head, laying out specific priorities and a 
timeline by which matching money had to be committed. “Letting the private partners only 
partner on funding of already selected ideas does not seem like much of a partnership,” said 
one grantmaker. “I think it runs the risk of narrowing the focus of private funding for 
education to those things the Department has cited as priorities,” agreed another. “This has 
the potential to limit greatly both the scope of innovations and the issues that need to be 
addressed across the very diverse education landscape.” Education Week quoted Chris 
Tebben, executive director of the affinity group Grantmakers for Education, as saying that the 
competition had highlighted a tension between funders’ desire to “leverage resources beyond 
philanthropy” and their need to “maintain [anj independent role ... and a perspective that is 
not limited to one [presidential] administration.” 46 

Moreover, this sequencing of the match at the end of the 
process meant that although they had advance warning of 
the timeline, funders would have just five weeks — from the 
notification of finalists on Aug. 5 to evidence of the match 
required by Sept. 8 — to determine which applicants to match, 
due to ARRA regulations mandating that funds be committed 
by Sept. 30. As school organizations ramped up for the 
coming school year and many donors and foundation boards 
went on traditional summer holidays, August was not an 
ideal time for such decisions to be made. “For the first time 
in a hundred years of education grantmaking, August was the 
busiest month,” noted one foundation executive. 

When the highest-rated applicants were announced, funders 
told us they faced two challenges: many funders were unsure 
of or disappointed by the names on the list (which could 
be termed “false positives”) and displeased by some of the names that were missing, whom 
they had hoped to fund (“false negatives”). Two foundation leaders estimated that less 
than 40 percent of the applicants that they were ready to provide a match for were selected 
as grantees — although given that less than 3 percent of applicants won, this is actually a 
significant overlap. Another funder said it provided no match at all because only one of the 
handful of applicants its staff hoped would win did so, and that organization did not need that 
foundation’s help on the match. Skepticism was most palpable in the Development category. 
“My excitement at the outset was finding the new exciting thing; this process didn’t surface 
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that,” noted the funder who didn’t provide matching funds to any of the i3 grantees. “Out 
of the Development grants, I would be amazed if these grantees really develop into game- 
changers,” noted another funder, who did provide some matching funds nonetheless. However, 
others reported they were excited to learn about new efforts and welcomed the opportunity to 
engage with organizations that were not yet on their radar. “We tried to remain open to new 
organizations and ideas that fit our criteria,” said one grantmaker. “We wanted to learn about 
new things, and it was pretty understandable that this would require more diligence than 
partnering with a group that we knew well.” “This broadened the universe of connections in 
the space and enabled philanthropy to learn about new projects and initiatives,” said another 
funder, despite the fact that her foundation didn’t end up providing matching funds. 

The goal of aligning investments around Department priorities, combined with the compressed 
timeline, led to some positive process innovations by philanthropists. “I’m excited about 
the way that i3 forced changes in the normal ways of doing business,” one interviewee told 
us. The program coordinated resources for specific attention to innovation in a way that 
simply hadn’t been done before, and pushed philanthropists to support innovations backed 
by evidence. “Many nonprofit organizations with proven results, such as America’s Choice, 
Aspire Public Schools, Citizen Schools, KIPP Schools, the New Teacher Center, the New 
Teacher Project, the Success for All Foundation, and Teach for America, get passed over for 
funding by large private foundations,” noted a recent Stanford Social Innovation Review 
article written by executives from the William and Flora Hewlett Foundation. “In fact, the 19 
nonprofits that ranked highest in the U.S. Department of Education’s Investing in Innovation 
(i3) competition, which was based on evidence of impact, on average had grants in 2010 from 
only three of the nation’s top 50 education foundations before winning the i3.” 47 The funders 
we interviewed generally agreed that the emphasis on evidence, particularly at the later Scale- 
Up stage, was important. “The evidence goal was huge,” said one interviewee. “It sparked 
very different conversations [with grantees] than the usual conversation of ‘we need money.’” 

The philanthropic funding market tends to be idiosyncratic and slow, and is often missing 
a sense of urgency among funders compared with the need by nonprofit managers and 
organizations. The very design of i3 influenced these behaviors, acting as a coordinating force, 
with the ARRA timeline adding an accelerating factor. Some funders organized new streams 
of work around the i3 matching opportunity, including the W.K. Kellogg Foundation, which 
provided $1.4 million in technical assistance to rural applicants and $4 million out of a total 
$11 million in a pooled match fund for rural organizations, and JPMorgan Chase, which 
worked hard to rally many individual donors to make matches. Foundations had to work 
far more quickly than their typical grant cycle, making final matching decisions in just over a 
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month (though many were already monitoring the scene throughout the spring and summer) — 
lightning speed compared to the quarterly or annual grant cycles at most foundations. Many 
foundations found ways to make these decisions quickly, such as requesting pre-approval 
for program staff to make final match decisions during the August window when boards of 
trustees would be out of reach. Those funders that did contribute matches found they had 
to work not only quickly, but also had to set up different decision-making processes. For 
example, one foundation told us that it had set aside funds from other non-education areas 
of the budget to use for i3 matches. Another secured the approval of its board in advance to 
allow just its education committee to approve i3 matches. Yet another scheduled a special 
meeting of its board to approve i3 matches in time for the September deadline. 

Despite the stress it induced, this sense of urgency was welcomed among some applicants. 

“It’s a fundraiser’s dream,” said one Scale-Up grantee. “There is an ultimate benefit, painful 
as it is, to having a shorter runway and forcing decision making.” Another interviewee saw 
some funders turn into development directors, “rolling up their sleeves and acting like board 
members to recruit others” by holding conference calls to help their grantees secure a match 
from other funders. Decisions came down to the wire, with the Department waiting until Sept. 
20 — nearly two weeks past the official deadline and just 10 days before stimulus funds had to 
be obligated — to announce that all 49 finalists had secured their match. 

Grantmakers themselves admitted that they appreciated the spur to action. “Funders were 
forced to operate on a short timeline, work together in new and different ways, and go 
beyond their comfort zone in ways that were really helpful to the sector,” said one funder we 
interviewed. “The manufactured crisis was helpful. It created a sense of urgency and urged 
people to take a leap of faith,” one grantmaker told us, adding that she would have still 
preferred more time to vet the grantees and connect them in-person with prospective donors 
to strengthen the connection (and perhaps boost the amount of money donors were willing to 
contribute). Likewise, other donors found they were able to work quickly, but still managed to 
squeeze in some of their usual diligence, including calls to applicants. “No one in the private 
sector would think it is too little time. There, people make decisions all the time in short 
order,” noted one foundation leader. “Foundations need to do what they can to create a more 
agile and nimble field. We have to build in this flexibility in philanthropy to be able to respond 
to these opportunities and not assume that existing processes are the only way or the only 
template in which everything else should fit. ” 

In our interviews, there was also general appreciation from philanthropic leaders for the 
opportunity to collaborate and leverage their work and resources, with some griping about 
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the process and timing. “Anything we can do to get traditional funders and newer ones to 
get out of their silos is good, and the process of drawing people out and creating a platform 
for them to work together was great,” said one funder we interviewed. To help facilitate the 
matching process, a collaborative of foundation leaders worked with the Bill & Melinda Gates 
Foundation in spring 2010 to create the Foundation Registry, a Web site where applicants 
could post abbreviated versions of their proposals for interested funders to sift through and 
review, and where funders could share diligence with one another about applicants and 
applications. Nearly 50 foundations signed up for the Registry, and nearly 700 applicants 
(less than half of the 1,700 total applicants) loaded summaries into the system. The site was 
developed by experienced product designer and manager Kartik Raghavan, an experienced 
Microsoft technologist recruited by the Bill & Melinda Gates Foundation with guidance from 
a collaborative of foundation leaders including Richard Laine of the Wallace Foundation, 
Barbara Chow of the William and Flora Hewlett Foundation, Stefanie Sanford of the Bill 
& Melinda Gates Foundation, and Leah Hamilton and Michele Cahill of the Carnegie 
Corporation of New York, who all saw the need to help foundation officials quickly review 
and assess a large group of prospective grantees and make informed matching decisions. 

Core Registry functionality included basic application searching and sorting by grant type, 
the “absolute priority” address and the date the application was uploaded. In addition to 
creating a short “common application” that saved grantees from having to submit multiple 
match requests, the Registry also featured the ability for foundations to mark “favorite” 
proposals and share notes about their impressions of applications or applicants. It is important 
to acknowledge that this type of shared diligence is rare in the philanthropic sector, where 
opinions and analysis by foundations are tightly guarded and almost never shared openly 
online. In at least one case, this feature led to one of the foundations we interviewed providing 
$350,000 in matching funds to an organization it had not funded before, based in large 
part on persuasive diligence notes uploaded by another foundation about the strength of the 
organization’s outcomes and evidence. 

The final press release announcing that all i3 grantees had secured their match showed that 
more than half had received some support from at least one of the Registry members 48 , but 
few would assert that the Registry was wholly responsible for all those connections. Still, 
a significant number of foundation staff we interviewed found the Registry very helpful. A 
local funder noted that the Registry was indispensable in identifying in short order all the 
applicants doing work in its target geography. Another funder noted that it made extensive use 
of the filters and categories to determine which applications would be worth reading in full. 
“The information was consistent for the various categories” which made it “easy to directly 
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Summary of i3 Matching Activity, 2010 



Type of Contribution 


Total Match 
($MM) 


Foundation 

Registry 

Members* 

($MM) 


Other Matching 
Funders ($MM) 


New Cash 


$101.6 


$48.1 


$53.5 


Repurposed Cash 


$9.1 


$7.2 


$1.9 


In-Kind (Not Applicant or 
Partner) 


$19.5 


$0.3 


$19.2 


In-Kind (Applicant or Partner) 


$8.3 


- 


$8.3 


Total 


$138.6 


$55.6 


$83.0 



Totals may not match due to rounding 

* Funding provided by members of the Foundation Registry, whether or not the funding occurred 
through the Foundation Registry. 

Source: U.S. Department of Education 



compare applications for [similar] projects,” this funder explained. It is worth noting that 
unlike peer reviewers, who were instructed not to compare similar applications, grantmakers 
naturally tended to do so, to ensure that their funds would go to the organization they thought 
was most promising within a given focus area. 

Besides providing funders with specific diligence support such as sorting capabilities, a short 
common application to review and shared diligence notes, the Registry seems to have been 
an important conversation-starter. “The Registry opened up conversations about how to 
reduce inefficiencies, such as through common application and reporting forms,” said one 
foundation executive. “The process of talking to each other has the potential to reduce the 
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cost of due diligence and introduce significant economies of scale. ” Some other funders felt the 
Registry was underused, and worried about security of confidential information, though the 
Registry creators assured us there was extensive security surrounding who could access what 
information. “The Registry didn’t seem confidential to the point where people could be really 
honest about their experiences with grantees,” said one foundation leader, who suggested the 
shared diligence might have been better suited to topical or regional phone calls. Still, most 
viewed the tool’s mere existence as a real step toward philanthropic collaboration and one that 
could be a springboard for future efforts. In the words of one funder, “technology should just 
be the starting point, and it is only as good as the conversation it drives.” 

While most funders appreciated the Registry, applicants generally didn’t give much credit 
to the Registry for mobilizing matching funds. In some ways this shouldn’t be surprising, 
given that the platform was designed largely to help donors, and provided little transparency 
for applicants into funders’ actions or progress. Applicants could see which funders were 
“considering match” but not any indication of real progress or how to follow up proactively. 
Rather than appreciating the fact that the Registry enabled them to provide only one common 
application for multiple funders to consider, many applicants instead viewed the Registry’s 
proposal format as an unfair additional amount of work beyond the federal application. 

Others believed (incorrectly, as it turned out) that if they already had a match cited in their 
application that they were disqualified from submitting their application to the Registry. 

(This was not the only area of confusion about matching funds for applicants. In one of the 
many ironies of this process, the Department tried to be more flexible and open about what 
“counted” as a match, but was inundated with so many questions about it from applicants 
who wanted to know exactly how to be in compliance that the agency added five rounds of 
frequently asked questions and answers to its Web site from March through July. Eventually, 
the Department even provided six sample match letters to illustrate the different types of 
acceptable matches.) 

Ultimately, i3 can claim credit for directing at least $140 million in private matching funds 
to the highest-rated applicants — but it is difficult to say for sure just how much of that was 
truly “new” money steered from other sectors or from other, less-effective educational efforts. 
Incomplete information about the breakdown of winning applicants’ matching grants — 
combined with a lack of clarity in some cases about whether those funders were “new,” or 
not — creates only a partial picture. The Department has reported that more than 250 different 
organizations provided more than 325 matching grants to the 49 i3 grantees. According to 
Department analysis, more than $100 million of the matching funding came in the form of 
new cash commitments to support the 49 projects, $20 million as in-kind donations, nearly 
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$10 million in repurposed cash, and the remaining $10 million as in-kind donations from the 
applicants or partners themselves. About half of that new cash came from foundations, and 
the rest was somewhat evenly divided among private companies, corporate foundations, and 
the applicants or partners themselves. 49 

However, full information about the details of each applicant’s match was not made available, 
especially the degree to which “new” commitments came from brand-new sources, from past 
donors or from hybrids of the two, such as donors that had given to local offices but never 
before to the main organization. In our (admittedly unscientific) sampling, the degree to which 
i3 mobilized new resources seems to have differed by stage. The match requirement seemed 
to bring in proportionately more “new” money to Development grantees, who may not yet 
have been on the radar of funders and for whom the visibility of i3 made a disproportionate 
impact. One Development stage grantee reported it raised more than 95 percent of its match 
from new sources, admitting, “The last $500,000 of that was the hardest money I’ve ever 
raised.” It is worth noting that many of these new donors for Development-level grantees are 
smaller foundations or involve individual donors. A number of interviewees hypothesized 
that perhaps the Development level was a sweet spot for mobilizing new donors: the size of 
the match required was more moderate, and the efforts were newer or less well known, and 
thus made new donors feel like they were “getting in on the ground floor” rather than just 
following larger national funders into supporting more-established organizations. 

At the Validation level, we talked with grantees who reported raising their matching funds 
strategically — some focusing on existing donors, while others purposefully focused on 
mobilizing new donors. “Because we did the bulk of fundraising locally, for the most part, 
we were able to work with partners who work with us on a regular basis for other reasons or 
extended what they would have done,” said one district Validation grantee, while a nonprofit 
Validation grantee told us it used i3 as an opportunity to widen its base of support and 
raised 40 percent of its match from new sources. At both the Validation and Scale-Up levels, 
it proved difficult for these large, established organizations to find truly “new” funders they 
had not already encountered over the years. Most of our interviewees at those levels told us 
that even their “new” monies came from donors they had already been courting, but that i3 
provided a forcing function that accelerated action. 

Besides the challenge of the compressed time frame to secure matching funds, it appears 
that the most significant lesson of the matching process is that it did not differentiate the 
appropriate percentage of private funding required for each of the stages of an innovation’s 
development. Many of our interviewees questioned the decision to require the same 20 percent 
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match across all stages of applicants. “There should be a sliding match scale: the greater 
the evidentiary base, the greater share that government should bear,” noted one funder. 
“Philanthropy typically can be most impactful in riskier situations,” agreed another funder. 
“The question here is if scale-up really needs the same attention and percentage of private 
dollars, versus those ideas that need it to get off the ground and have a chance to develop.” (It 
is worth noting that the second round of i3 does differentiate the match required at each stage, 
with just a 5 percent match required at the Scale-Up level, 10 percent for Validation, and 15 
percent for Development; see page 56 for more on the program’s second round.) 

Some suggested that the public-private partnership was most successful in engaging smaller 
donors and putting innovative efforts squarely on their radar. The individual donors and 
local foundations we spoke with appreciated the opportunity to learn from other, larger 
players through the process. “We not only had the benefit of being forced to prioritize 
collectively, but also we had the opportunity to work with national funders in a way we do 
not often do and to learn from them,” said one local foundation representative. Similarly, 
an interviewee who aggregates individual donations said her donors liked the idea of the 
federal government leveraging their dollars and the way i3 asked the field to put its best 
ideas forward. Some applicants suggested that, given the difficulty of getting large national 
foundations to move beyond their established focus areas (by subject or geography), it might 
even make sense for these smaller, newer education funders to be an explicit target of future 
matching requirements, along with other funders that don’t have dedicated education giving 
programs, like corporate foundations. “National foundations have multi-year strategies and 
grants, and it doesn’t seem to make sense for them to have funds diverted to a match,” said 
one philanthropic interviewee. “Individual donors can get to know grantee organizations well 
over the next three to five years through a process like this, and then help them expand their 
network of individual supporters.” 

Finally, with respect to philanthropic funding, it remains unclear whether the onetime 
matching funds directed toward i3 grantees will translate into longer-term improvements 
in support for education innovation and evidence-based improvements. For example, the 
Department of Education has made several efforts to highlight the unfunded “highly rated” 
applicants — those applications receiving a score of 80 or greater — especially at the Scale-Up 
level, in order to mobilize what the agency refers to as “secondary” funding. A section of 
the Department’s Web site shares those project narratives and their peer review scores, and a 
January 2011 event with the Aspen Institute on education innovation included, in addition to 
panels on innovation investing in education, an “expo” to showcase these promising concepts 
to prospective donors and investors. 50 Several applicants told us they did not feel the “expo” 
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led to productive donor activity, and they are skeptical about whether the secondary funding 
efforts will lead to any new money. Many non-grantees worry that the projects they spent 
months putting together will fail to secure any traction or additional funding. The funders 
we surveyed were more optimistic, with nearly a quarter saying they have provided funds to 
applicants that did not win (including those that had established dedicated pools of funding 
to use on i3 matches that were not spent down during the matching process), while others 
are still considering whether to do so. As in the analysis of whether “new” funding flowed to 
i3 grantees, it is unclear whether these funders would have made these commitments in the 
absence of i3. 

One final issue about public-private collaboration that came up repeatedly in our interviews 
was the absence of for-profit applicants and investors. There was general appreciation of 
why it might have been difficult to include for-profit applicants. However, many of our 
interviewees — including applicants, funders and observers alike — emphasized that in order to 
support the kind and scale of innovation required to transform public education, the financial 
and talent resources of the private sector must be tapped. The i3 fund was set up in a way that 
constrained most matches to the philanthropic sector, with the private sector participating 
only through corporate philanthropy or in-kind donations. Allowing innovative for-profit 
companies to enter the competition would have invited matching funds and greater interest 
from the vast field of for-profit investment capital. This was particularly problematic with 
regard to technology-enabled innovations: with so many qualified for-profit players shut out of 
the process, it is no surprise that there were few applications of technology to the innovations 
proposed by i3 applicants and grantees. 
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KEY TAKEAWAYS, RECOMMENDATIONS AND CONCLUSION 



The Investing in Innovation program was the first of its kind in education. As such, it 
represents a potential object lesson about what happens when a massive government 
bureaucracy best known for mandating compliance and disbursing formula funds attempts 
to reorient itself — and the very ecosystem around it — toward increased innovation among 
practitioners, improved evidence-gathering, and better outcomes for students and communities 
at scale. 

The experiment is still under way with current i3 grantees, who will be joined later this 
year by a new crop of grantees from the program’s smaller ($150 million) second round. 

In addition to carrying out the actual work behind their funded projects, initial i3 grantees 
will also be convened at least annually by the Department for project directors meetings to 
compare notes, conduct evaluations of their work (and at the Validation and Scale-Up levels, 
share their data sets with third-party researchers) and find ways to document and share their 
efforts through communities of practice and other means. The Institute of Education Sciences 
has contracted with Abt Associates to conduct a five-year, $9.4 million evaluation of the first 
round of the i3 program itself, including a meta-analysis of the individual grantee evaluations. 
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It is premature to assess the ultimate impact of the i3 program, its grantees and the 
innovation ecosystem writ large. However, it’s not too soon to learn from i3 as an attempt 
at organizational and market innovation initiated by the public sector. 



Key Takeaways 

As we noted throughout this paper, our analysis of the design and initial effects of i3 showed 
areas of real progress and real challenge. 

Progress: 

» Focus of national attention on the need for innovation in education; 

» Emphasis on scaling up what works to address the country’s significant educational 
problems; 

» Introduction of a graduated evidence framework that tied federal investments to impact 
and allocation of greater resources toward those who met the most rigorous evidence bar 
for impact on student outcomes; and 

» Steering and accelerating resources toward a specific set of investment priorities aligned 
with important emerging demand in the field — and doing so in a more transparent, 
collaborative and evidence-based way than is typical of the federal government. 

Challenges: 

» Narrow eligibility requirements that shut out new or very early-stage organizations and 
nearly all for-profit providers (and their investors); 

» A limited definition of acceptable evidence that skewed and constrained the potential 
applicant pool in significant ways; 

» An over-simplified process inadequate to the complicated task of selecting emerging, 
promising and proven innovations; and 

» A timeline that left little room for meaningful diligence. 

At its heart, the i3 program had two important goals that were fundamentally in tension: 
“innovation,” which implies new ways of doing things; and “scale,” in which things that 
have been demonstrated to work are replicated and disseminated. To create real change, both 
innovation and scale are required, but implementing them well requires different decision- 
making and support systems, as well as different people and matching processes. For example, 
requiring a “proven track record” may have shut out applicants that could have delivered 
more truly break-the-mold innovations. And, as we will explain further below, scale requires 
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cautious assessment of demonstrated evidence and relatively little risk-taking, and as such may 
be well-suited to the public sector, while early-stage innovations often require more intuitive 
assessment of opportunity and future directions, and greater risk tolerance to support new or 
emerging organizations, both of which may lend themselves more naturally to the private and 
philanthropic sectors. 

Moreover, while scale matters and it is important to invest funds to get there, using education 
innovation in a smart way to urge the system toward performance-based practices will require 
better assessments of productivity and return-on-investment (ROI) for the funds that are 
invested — including public funds. In the simplest terms, ROI would be the cost per unit of 
impact. The original i3 requirement to specify the cost of the innovation on a per-student- 
served basis and to encourage scale (by asking applicants to specify the cost to scale to serve 
hundreds of thousands of students, even if they didn’t actually intend to) began to move in this 
direction; a competitive preference in the program’s second round awarding a bonus point to 
projects that improve productivity (see page 56) may represent another small step forward. 

The Department could lead the field forward significantly by assessing ROI across different 
kinds of programs and interventions at early and later stage or scale — and incorporating 
such metrics into grant decisions. Certainly, ROI is a complicated notion in a complex field 
like education where diverse outcomes are often difficult to prioritize (from reading on grade 
level by third grade, to on-time high school graduation, to mastery of critical thinking and 
problem-solving skills that are critical for success in college and life). It won’t be easy to 
assess trade-offs between investments in programs and interventions that can reach many 
more students versus those that can accomplish deeper or broader kinds of impact for fewer 
students. But as progress continues toward a more outcomes-driven education system while 
holding educational spending steady (or even decreasing it), it would be worthwhile to have 
real outcomes and cost data to inform that policy debate. 

In addition, optimizing the participation of all the sectors — government, private and 
philanthropic — in education innovation is necessary, even though it is difficult and 
controversial. The designers of the i3 program may not have gotten all the details right here, 
but they were on the right track: the government must join with the nonprofit and business 
sectors to help create an ecosystem that embraces a continual learning cycle and effectively 
taps robust innovation engines to deliver constantly improving outcomes for all students. 

This means both that the private sector must be involved and that the government must play 
a crucial role in defining clear acceptable outcomes and holding all three sectors — including 
public agencies and nonprofits, too — accountable for outcomes aligned with the public 
good. To do this well will require: clearer metrics for the desired educational outcomes; 
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better and deeper assessments and effective indicators of progress along the way; and useful 
accountability mechanisms that open up the process to more providers (and investors) while 
also ensuring that their efforts make a real difference for the students and communities this 
public agency is charged with serving. 

Likewise, just as different learners require differentiated instruction, it is vital to differentiate 
selection and support for the various types and stages of innovation that receive public 
funding. While the differentiated evidence levels and scoring rubrics of i3 were a very 
important step in the right direction, the various stages of innovation need even more distinct 
approaches to due diligence, investment decision-making, private sector match, and ongoing 
support and accountability. In particular, it seems clear that the traditional peer review 
process is ineffective for assessing early-stage innovations, and that requiring one level of 
matching funds across multiple stages of applicants failed to optimize the way the private and 
philanthropic sectors could support innovation. (Of these, the level of private sector match 
appears to be the only element that was differentiated further in the program’s second round.) 



Recommendations for the Department of Education 
Investing in Innovation 

Some of the immediate lessons of the first round of i3 have influenced the design of its smaller 
$150 million second round, which was announced in June 2011, with applications due August 
2 and grant awards to be made by year-end. The new round drops the absolute priority 
focused on data, and adds two new absolute priorities, one focused on rural schools (which 
replaces the similar competitive priority there) and another focused on science, technology, 
engineering and mathematics (STEM) education; applicants choosing the rural priority are 
also “encouraged” to address one of the other four priorities. Competitive preferences related 
to improving productivity (“projects that are designed to significantly increase efficiency in 
the use of time, staff, money, or other resources while improving student learning or other 
educational outcomes”) and technology (“projects that are designed to improve student 
achievement 4 or teacher effectiveness through the use of high-quality digital tools or 
materials, which may include preparing teachers to use the technology to improve instruction, 
as well as developing, implementing, or evaluating digital tools or materials”) were also 
added, though applications can only receive points for up to two such preferences. In a nod to 
the differing levels of support required from the private sector at different stages, the percent 
of matching funds required will be scaled, with Development grantees required to secure a 15 
percent match, Validation grantees 10 percent, and Scale-Up grantees just 5 percent. However, 
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the program will continue to rely on peer reviewers - though they will not assign scores to 
the actual evidence, leaving that for the review of the Institute of Education Sciences — and 
will still limit eligible applicants to LEAs alone or in groups, or nonprofits in partnership with 
LEAs. 51 

While we applaud these changes, we recommend that the Department consider going even 
further, embracing the need for greater differentiation across all elements of its innovation 
grantmaking process: using different approaches according to the unique needs of different 
priority areas; continuing to differentiate the types of evidence required based on the maturity 
and comprehensiveness of an innovation; supporting different kinds of innovations in ways 
tailored to their needs; and engaging the private sector to optimize the strength and scale 
of innovations and their impact on the field. This might mean, for example, supplementing 
direct grants with incentives that would mobilize more risk-tolerant funders — such as small 
foundations and individual donors - in support of public goals, and possibly even adding 
private-sector incentives like tax breaks for angel investors that back companies tackling 
important educational priorities or accomplishing important educational goals. (Some of this 
private-sector work could be done through the Digital Promise effort rather than i3). 

As for supporting specific stages of innovation, there was broad consensus that the i3 
program’s eligibility requirements, peer review process and rigid confines of government 
decision-making together may have been ill-equipped for identifying the most promising early- 
stage innovations. We agree with those we interviewed who proposed that the best way for 
the federal government to support early-stage innovation might be to provide incentives or 
to direct funds to expert intermediaries who are more inclined to spot promising innovations 
and embrace and manage their risks; these intermediaries could then invest in both nonprofit 
and for-profit companies that fit the bill. Some interviewees hypothesized that the best use 
of future rounds of i3 funding might be at the Validation stage, where the flexibility and risk 
tolerance of the private sector had brought a promising new idea to a point where larger-scale 
government resources would be well spent to build and assess a stronger evidence base and 
lay the foundation for potentially broader scale. Most also felt there was an important role for 
government to play in scaling up successful programs, but there was genuine and widespread 
confusion about how to accomplish that given how out of favor earmarks have become. 

We recommend that the Department continue its path toward greater differentiation of 
support across the types and stages of innovation (including field scans and intentional 
development) as well as throughout the life cycle of innovation, increasing the ways that the 
process, metrics and supports are differentiated by stage. The following table outlines one 
possible way to think about some of the salient points of differentiation for future efforts. 
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Recommended Approaches to Federal Funding of Education Innovation 





Development Stage 


Validation Stage 


Scale-Up Stage 


Goal 


Mobilize new activity 


Provide funds to 


Provide funds to 




and experimentation 


organizations with 


programs that 




around specified 


promising results 


have demonstrated 




problems of practice 


that could be game- 


significant impact in 




from the field, and 


changing, and prepare 


multiple locations but 




identify disruptive 


those solutions for 


need growth capital to 




solutions not yet under 
development 


greater scale by testing 
them for efficacy 
across more contexts 
and by investing in 
more robust evidence 
evaluation 


reach optimal scale 


Features 


Limited track record; 


Some track record of 


Long track record 




direct track record often 


success with specific 


of success and well- 




inversely correlated to 
novelty of innovation 


innovation 

May be organizations 


developed evidence 
base, including 
multiple independent 




New and existing 
organizations with new 
programs 

Consider including 
for-profit players to 
help steer them toward 
priority impacts 


and programs 
emerging from earlier- 
stage pipeline, or new 
entrants that surface 


evaluations 
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Development Stage 


Validation Stage 


Scale-Up Stage 


Evidence 


Identify criteria for 


Require clear evidence 


Require demonstrated 




success and use 


of success and 


success through wide 




judgement to establish 


provide funding for 


range of studies — 




appropriate interim 


experimental or quasi- 


including randomized 




indicators 


experimental studies 


controlled studies — that 
demonstrate efficacy in 




Assess and learn 


Consider alternative 


multiple contexts 




from efforts in other 


approaches to 






agencies such as the 


gathering evidence 


Require ongoing 




Department of Energy, 


more amenable to 


evaluation and 




National Institutes 


holistic/comprehensive 


assessment of results in 




of Health, National 
Science Foundation 
and Defense Advanced 
Research Projects 
Agency about how 
to allow flexibility on 
means for tracking 
indicators of success 


interventions 


new contexts 


Decision 


Screening of team's 


Interviews and site 


Compare multiple 


Making 


ability to take and 


visits to assess 


types of similar 




manage relevant risks 


capacity of team and 


interventions to 




as well as specific 


organization 


compare outcomes and 




relevant skill sets 




return on investment 






Qualified peer 


(ROD 




Decision makers must 


reviewers without 






be able to exercise 


direct conflicts of 


Decisions may be 




judgment based on 


interest may serve as 


made by peer review 




industry expertise; 


advisors, particularly to 


(combination of content 




not well-suited for 


assess evidence and 


and evidentiary experts) 




standardized rubrics or 


contribute content 


with appropriate 




peer review 


expertise 


screening for direct 
conflicts of interest 




Work through a set of 


Public and private 






qualified independent 


sector representatives 






intermediaries 


should be involved 
to optimize both 
public good and 
organizational 
development 
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Development Stage 


Validation Stage 


Scale-Up Stage 


Investment 


Goal: Use public 


Goal: Use public and 


Goal: Ensure proven 


Funding 


funding to stimulate 


private/philanthropic 


innovations meet the 


Mix 


investment in 


resources together 


scale of the need 




priority outcomes/ 


to advance evidence- 






areas by private and 


gathering for 


Proven nonprofit and 




philanthropic sectors 


promising innovations 


for-profit innovations 
should be supported 




Private and 


Much of the early- 


largely by revenues 




philanthropic sectors 


stage risk has been 


(state, local and federal 




more nimble, risk- 


removed, qualifying the 


dollars, ideally with 




tolerant and able to 


organization for greater 


policies that incent or 




apply judgment, so 


public investment, 


require some percent 




more than 50% should 


which can leverage 


of formula funds to 




come from private and 


philanthropic capital 


be used on proven 




philanthropic sectors in 


(where funding for scale 


programs) 




most cases 


is limited) 


Additional growth 




To maximize 


Robust public-private 


investment capital 




participation 


partnership should 


should also be available 




from private and 


capitalize on public 


to nonprofits only; 




philanthropic sectors, 


sector research/ 


for-profits have access 




include both for- 


evaluation capacity and 


to late-stage private 




profit and nonprofit 


private/philanthropic 


equity funds 




organizations 


sector organization- 
building expertise 


Public share of funding 




Examples: the Small 




for additional growth 




Business Administration 


Public share of funding 


and R&D should be 




offers 1:1 matching 


for nonprofits at this 


as high as 90% for 




of private funding 


stage should be at least 


nonprofits, as public 




for qualified small 


50% 


sector bears little 




businesses; Social 




risk (but significant 




Innovation Fund 


Public share of funding 


potential social good) 




required 1:3 matching 


for for-profits minimal 


at this stage 




of federal dollars to 


at this stage — should 






private or philanthropic 


be able to access 






resources 


growth capital with 
proven product — and 
reserved for cases 
where research would 
be broadly shared and 
inform the wider market 
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Development Stage 


Validation Stage 


Scale-Up Stage 


Support 


Active, hands- 


Focus public support 


Require some percent 




on, iterative, from 


on nonprofit capacity 


of public formula funds 




experienced 


for scale and on 


(such as Title 1) to be 




practitioners 


strengthening evidence 


used on programs with 






base 


this kind of proven 




Define clear milestones 




impact at scale, which 




for progress and exit 


Focus philanthropic 


will provide incentives 




quickly if insufficient 


support on 


and rewards for 




progress 


organization-building 


effective programs 






and talent recruitment 


and products across 
nonprofit and private 








sectors. 



Source: Bellwether Education Partners 



Public-Private Partnerships 

Advancing innovation in public education will require the involvement of the public sector 
and the private sector, including the philanthropic and nonprofit communities. The proposed 
ratios of funding contributions laid out in the table above emphasize that the private sector 
can and should take on greater risk in the earlier stages of education innovation, and that 
the burden of support should be shifted toward the public sector as products, services and 
approaches prove their merit for serving a larger scale of the public and therefore deserve 
more funding from public sources. 

At the Validation level, for example, innovations need capital for both program expansion 
(traditionally the bailiwick of the more nimble and experienced private and philanthropic 
sector) and for robust evaluation and efficacy research (where the government’s objective 
research capacity and greater resources may excel). 

Many would argue that programs that meet the high bar for evidence found at the Scale-Up 
level should even be 100 percent publicly funded, as they better accomplish public goals than 
many status quo programs and should be allowed to access public funds to meet student 
needs, thereby insuring a higher ROI to the public for tax dollars spent. Some have suggested 
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that perhaps one way to do this would be by steering funds that are currently allocated on 
a formula basis (like Title I, the piece of the Elementary and Secondary Education Act that 
directs federal funds to schools serving high-need students) toward programs that have 
demonstrated effectiveness. For instance, if 10 percent of the roughly $14 billion in annual 
Title I funds were allocated in this way, $1.4 billion could more effectively fund scale and shift 
Title I funding toward a more outcomes-driven approach. As a publication by the Center for 
American Progress so astutely put it in a memo arguing for more competitive federal funding, 
“Activities that lead to improved educational outcomes and results could be identified and 
rigorously evaluated, and future spending across major formula grant education programs 
could then help support such reforms across all high-poverty schools.” 52 As we have addressed 
in previous papers 53 , nonprofit organizations in particular struggle to raise capital to grow 
their programs to scale, in part because of the quirks of the philanthropic capital market 
and also because of the way nonprofit growth funding tends to have to be categorized as 
revenue rather than as equity. If larger scale revenues were provided through formula funds 
for effective organizations, such organizations could use philanthropic capital and competitive 
grants like i3 to grow and scale their organizations more effectively. 

Other ways to support these programs include efforts like Race to the Top, which provide 
the demand side (particularly buyers like states and districts, who act on behalf of users like 
teachers, students and families) with incentives to measure, track and reward outcomes, thus 
increasing the likelihood that effective innovations will be adopted. 

One final way of optimizing the role for each sector in a public-private partnership for 
education innovation deserves special mention here: allowing for-profit organizations — or 
at least investment intermediaries that can support for-profit investments — to enter the ring. 

This would enable the federal government to tap some of the extraordinary innovations 
taking place in the private sector for the good of public schools, including adaptive learning 
technologies and productivity tools from other industries that could be specialized for 
educators (the way In-Q-Tel invests in emerging technologies in service to the national security 
community), not to mention opening the field up to the vast financial and talent resources that 
sit in the private sector. 



Recommendations for Grantmakers 

As for philanthropy, donors need to go beyond simply keeping the Foundation Registry alive 
as a tool for i3 grantees. There is room for a great deal of process innovation in the way 



Bellwether Education Partners 63 



grantmakers work together in application, diligence and reporting, and tools like the Registry 
may be useful in moving toward common application and reporting tools that can minimize 
the compliance and fundraising burden on nonprofit innovators. 

While attempting to align donor efforts with desired outcomes and to standardize some 
of their processes to lighten the load on nonprofit managers, it is also valuable to consider 
ways of better differentiating among segments of donors to optimize their support for 
different stages of innovation. Individual donors, local or regional foundations, large national 
foundations and corporate philanthropy all have different levels of risk tolerance and unique 
value to add, and we should be more explicit about these choices and priorities in order to 
maximize their impact on individual organizations and on the field of education innovation. 
Each of these segments may also require different tools and types of information to make 
decisions (some more data driven and others more intuitive), and it is worth considering 
whether donors in different categories may be better suited to different stages in the 
innovation cycle than others. 

Finally, philanthropists should also continue to support other forms of evidence beyond 
the randomized controlled studies emphasized by IES and the federal government, which 
are useful but very expensive and relevant only to certain kinds of interventions. Education 
innovation will also require more holistic approaches that will demand alternative ways to 
build a rigorous and compelling evidence base, and philanthropy needs to continue to invest 
in this kind of broad-based evidence building, as well as in knowledge-sharing tools and 
platforms that can inform the field better than do government clearinghouses. 



Conclusion 

Ultimately, i3 will be most successful if it leads not only to specific improvements in the 
amount and quality of education innovation, but also to changes in the ecosystem for 
educational innovation and continuous improvement — and to changing the perception of 
education innovation in the minds of voters and policymakers who can either support risk- 
taking in the interest of student achievement, or shy away from it in favor of continued 
mediocrity. In the words of one of our applicant interviewees, “Unless public demand says that 
we need to shift the way dollars are spent, we’re not going to change anything. ” As another 
interviewee reminded us, this is part of a much larger cultural shift for the Department and 
the field: “Whether i3 continues in new rounds or not, this is our shot to be intentional 
about continuing the conversation and pressing for a way of thinking that gets us to ask even 
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tougher questions, to create a more friendly environment for innovation and allocate resources 
to support innovation, both on the front end and to scale.” 

To get there, it will be important for both the Department and philanthropy to invest in 
ways to identify and disseminate useful information and lessons from the first pool of i3 
grantees, and to also help fix the broken demand side of the education innovation equation 
by encouraging and funding broader adoption of programs and tools that work. In addition 
to the second and any future rounds of i3, the Department should continue to seek out other 
public-private partnership efforts that direct funding toward promising solutions, toward 
evidence of outcomes and toward scale. Together, these values should become more often and 
more deeply embedded in the ways government works to align public and private resources 
alike toward better outcomes for students. 
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